Insights on Crosscoder Model Diffing
We report some developing work on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. We'd ask you to treat these results like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper. In this update, we investigate an unexpected phenomenon in crosscoder model diffing : features that are exclusive to one model tend to be more polysemantic and dense in their activations, making them difficult to interpret. Through experiments with toy models, we show that this likely emerges from competition for limited feature capacity – since shared features can explain neuron activation patterns in both models, exclusive features must encode more information to justify their allocation. We propose a mitigation strategy which introduces a small set of designated shared features with a reduced sparsity penalty, rendering the exclusive features more interpretable and mon
Insights on Crosscoder Model Diffing Transformer Circuits Thread Insights on Crosscoder Model Diffing Siddharth Mishra-Sharma, Trenton Bricken, Jack Lindsey, Adam Jermyn, Jonathan Marcus, Kelley Rivoire, Christopher Olah, Thomas Henighan We report some developing work on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. We'd ask you to treat these results like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper. Introduction and summary In this update, w
Explore this link on the map →saved by
related reading
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- Transformer Circuits Threadtransformer-circuits.pub
- On the Biology of a Large Language Modeltransformer-circuits.pub
- Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
- What We Learned Trying to Diff Base and Chat Models (And Why It Matters) — LessWronglesswrong.com
- Sparse Crosscoders for Cross-Layer Features and Model Diffingtransformer-circuits.pub
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
- An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
- Composer2.pdfcursor.com
- Stage-Wise Model Diffingtransformer-circuits.pub
- Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org