Paper Summary: Interpreting Language Model Parameters
Language models are some of the most remarkable computer programs in existence. They implement algorithms humans have tried and failed to write by hand for decades. Yet the "code" these algorithms are implemented by is "neural code", which is written, somehow, as enormous, inscrutable matrices of parameters. As a result, the field of interpretability has focused mainly on trying to understand the models' activations — the models’ “thoughts”. But reading these thoughts doesn’t immediately explain the computations that gave rise to them. To understand them deeply, we should understand not just the inputs and outputs to their computations, but the computations themselves. AdVersarial Parameter Decomposition[1] (VPD) is our technique for doing this in a language model. By splitting the model's parameters into simple, understandable pieces, we are able to directly study their structure and the algorithms they implement. We used VPD to decompose the weight matrices in a 67M-parameter languag
Language models are some of the most remarkable computer programs in existence. They implement algorithms humans have tried and failed to write by hand for decades. Yet the "code" these algorithms are implemented by is "neural code", which is written, somehow, as enormous, inscrutable matrices of parameters. As a result, the field of interpretability has focused mainly on trying to understand the models' activations — the models’ “thoughts”. But reading these thoughts doesn’t immediately explain the computations that gave rise to them. To understand them deeply, we should understand not…
saved by
related reading
- Interpreting Language Model Parametersgoodfire.ai
- Training Language Models to Explain Their Own Computationsarxiv.org
- Interpreting Language Model Parametersarxiv.org
- [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
- Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decompositionarxiv.org
- Towards Scalable Parameter Decompositiongoodfire.ai
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- Attribution-based parameter decomposition — LessWronglesswrong.com