Attribution Patching: Activation Patching At Industrial Scale — Neel Nanda
A write-up of an incomplete project I worked on at Anthropic in early 2022, using gradient-based approximation to make activation patching far more scalable
Attribution Patching: Activation Patching At Industrial Scale Feb 4 Written By Neel Nanda The following is a write-up of an (incomplete) project I worked on while at Anthropic, and a significant amount of the credit goes to the then team, Chris Olah, Catherine Olsson, Nelson Elhage & Tristan Hume. I've since cleaned up this project in my personal time and personal capacity. TLDR Activation patching is an existing technique for identifying which model activations are most important for determining model behaviour between two similar prompts that differ in a key detail I introduce a technique ca
saved by
related reading
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- Actually, Othello-GPT Has A Linear Emergent World Representation - Neel Nandaneelnanda.io
- [2404.15255] How to use and interpret activation patchingarxiv.org
- An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
- In-context Learning and Induction Headstransformer-circuits.pub
- Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
- An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
- Towards Automated Circuit Discovery for Mechanistic Interpretabilityarxiv.org
- Dreaming Vectors: Gradient-descented steering vectors from Activation Oracles and using them to Red-Team AOs — LessWronglesswrong.com
- Softmax Linear Unitstransformer-circuits.pub
- Attribution-based parameter decomposition — LessWronglesswrong.com
- Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io