Paper Summary: Interpreting Language Model Parameters
Language models are some of the most remarkable computer programs in existence. They implement algorithms humans have tried and failed to write by hand for decades. Yet the "code" these algorithms are implemented by is "neural code", which is written, somehow, as enormous, inscrutable matrices of parameters. As a result, the field of interpretability has focused mainly on trying to understand the models' activations — the models’ “thoughts”. But reading these thoughts doesn’t immediately explain the computations that gave rise to them. To understand them deeply, we should understand not just the inputs and outputs to their computations, but the computations themselves. AdVersarial Parameter Decomposition[1] (VPD) is our technique for doing this in a language model. By splitting the model's parameters into simple, understandable pieces, we are able to directly study their structure and the algorithms they implement. We used VPD to decompose the weight matrices in a 67M-parameter languag
Language models are some of the most remarkable computer programs in existence. They implement algorithms humans have tried and failed to write by hand for decades. Yet the "code" these algorithms are implemented by is "neural code", which is written, somehow, as enormous, inscrutable matrices of parameters. As a result, the field of interpretability has focused mainly on trying to understand the models' activations — the models’ “thoughts”. But reading these thoughts doesn’t immediately explain the computations that gave rise to them. To understand them deeply, we should understand not…
saved by
related reading
- Training Language Models to Explain Their Own Computationsarxiv.org
- Interpreting Language Model Parametersgoodfire.ai
- [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
- Transformer Circuits Threadtransformer-circuits.pub
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- Language models can explain neurons in language modelsopenaipublic.blob.core.windows.net
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- Tracing the thoughts of a large language model \ Anthropicanthropic.com
- Attribution-based parameter decomposition — LessWronglesswrong.com
- On the Biology of a Large Language Modeltransformer-circuits.pub
- Towards Scalable Parameter Decompositiongoodfire.ai
- Training Language Models to Explain Their Own Computationsarxiv.org