Insights on Crosscoder Model Diffing
We report some developing work on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. We'd ask you to treat these results like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper. In this update, we investigate an unexpected phenomenon in crosscoder model diffing : features that are exclusive to one model tend to be more polysemantic and dense in their activations, making them difficult to interpret. Through experiments with toy models, we show that this likely emerges from competition for limited feature capacity – since shared features can explain neuron activation patterns in both models, exclusive features must encode more information to justify their allocation. We propose a mitigation strategy which introduces a small set of designated shared features with a reduced sparsity penalty, rendering the exclusive features more interpretable and mon
Insights on Crosscoder Model Diffing Transformer Circuits Thread Insights on Crosscoder Model Diffing Siddharth Mishra-Sharma, Trenton Bricken, Jack Lindsey, Adam Jermyn, Jonathan Marcus, Kelley Rivoire, Christopher Olah, Thomas Henighan We report some developing work on the Anthropic interpretability team, which might be of interest to researchers working actively in this space. We'd ask you to treat these results like those of a colleague sharing some thoughts or preliminary experiments for a few minutes at a lab meeting, rather than a mature paper. Introduction and summary In this update, w
saved by
related reading
- Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org
- Weight-Sparse Circuits May Be Interpretable Yet Unfaithful — LessWronglesswrong.com
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- Toy Models of Superpositiontransformer-circuits.pub
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- Transformer Circuits Threadtransformer-circuits.pub
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
- Bridging the Attention Gap: Complete Replacement Models for Complete Circuit Tracinginterp.open-moss.com
- Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
- Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub