[2002.09815] Neuron Shapley: Discovering the Responsible Neurons

We develop Neuron Shapley as a new framework to quantify the contribution of individual neurons to the prediction and performance of a deep network. By accounting for interactions across neurons, Neuron Shapley is more effective in identifying important filters compared to common approaches based on activation patterns. Interestingly, removing just 30 filters with the highest Shapley scores effectively destroys the prediction accuracy of Inception-v3 on ImageNet. Visualization of these few critical filters provides insights into how the network functions. Neuron Shapley is a flexible framework and can be applied to identify responsible neurons in many tasks. We illustrate additional applications of identifying filters that are responsible for biased prediction in facial recognition and filters that are vulnerable to adversarial attacks. Removing these filters is a quick way to repair models. Enabling all these applications is a new multi-arm bandit algorithm that we developed to efficiently estimate Neuron Shapley values.

Neuron Shapley: Discovering the Responsible Neurons Amirata Ghorbani James Zou∗ Department of Electrical Engineering Department of Biomedical Data Science Stanford University Stanford University arXiv:2002.09815v3 [stat.ML] 13 Nov 2020…

related reading

• Zoom In: An Introduction to Circuitsdistill.pub
• Toy Models of Superpositiontransformer-circuits.pub
• Explaining Neural Network Models with SHAP Values: A Mathematical Perspective | by Kevin Akbari | Mediumakbarikevin.medium.com
• The Building Blocks of Interpretabilitydistill.pub
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• Language models can explain neurons in language modelsopenaipublic.blob.core.windows.net
• Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• Attribution Patching: Activation Patching At Industrial Scale - Neel Nandaneelnanda.io
• Neuronpedianeuronpedia.org
• Structure and Interpretation of Deep Networkssidn.baulab.info