[2404.15255] How to use and interpret activation patching
Abstract:Activation patching is a popular mechanistic interpretability technique, but has many subtleties regarding how it is applied and how one may interpret the results. We provide a summary of advice and best practices, based on our experience using this technique in practice. We include an overview of the different ways to apply activation patching and a discussion on how to interpret the results. We focus on what evidence patching experiments provide about circuits, and on the choice of metric and associated pitfalls.
How to use and interpret activation patching Stefan Heimersheim Neel Nanda stefan.heimersheim@gmail.com arXiv:2404.15255v1 [cs.LG] 23 Apr 2024 Abstract Activation patching is a popular mechanistic interpretability technique, but has many subtleties…
related reading
- Attribution Patching: Activation Patching At Industrial Scale - Neel Nandaneelnanda.io
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- Towards Automated Circuit Discovery for Mechanistic Interpretabilityarxiv.org
- An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
- Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org
- Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
- An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
- Interpreting Language Model Parametersgoodfire.ai
- Sparse Attention Post-Training for Mechanistic Interpretabilityarxiv.org
- Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
- Causal Scrubbing: a method for rigorously testing interpretability hypotheses [Redwood Research] — LessWronglesswrong.com
- An Introduction to Exemplar Partitioning for Mechanistic Interpretability — LessWronglesswrong.com