Training Language Models to Explain Their Own Computations

Can language models (LMs) learn to faithfully describe their internal computations? Are they better able to describe themselves than other models? We study the extent to which LMs’ privileged access to their own internals can be leveraged to produce new techniques for explaining their behavior. Using existing interpretability techniques as a source of ground truth, we fine-tune LMs to generate natural language descriptions of (1) the information encoded by LM features, (2) the causal structure of LMs’ internal activations, and (3) the influence of specific input tokens on LM outputs. When trained with only tens of thousands of example explanations, explainer models exhibit non-trivial generalization to new queries. This generalization appears partly attributable to explainer models’ privileged access to their own internals: fine-tuning a model to explain its own computations generally works better than fine-tuning a different model (even if the explainer model is significantly more cap

Zifan Carl Guo Affiliation: MIT CSAIL Vincent Huang Affiliation: Transluce Jacob Steinhardt Affiliation: Transluce Jacob Andreas Affiliation: Transluce Affiliation: MIT CSAIL Abstract Can language models (LMs) learn to faithfully describe their internal computations? Are they better able to describe themselves than other models? We study the extent to which LMs’ privileged access to their own internals can be leveraged to produce new techniques for explaining their behavior. Using existing interpretability techniques as a source of ground truth, we fine-tune LMs to generate natural…

saved by

• Dillon Nguyen

related reading

• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• Training Language Models to Explain Their Own Computationsarxiv.org
• Transformer Circuits Threadtransformer-circuits.pub
• Tracing the thoughts of a large language model \ Anthropicanthropic.com
• On the Biology of a Large Language Modeltransformer-circuits.pub
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Language models can explain neurons in language modelsopenaipublic.blob.core.windows.net
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• [2505.17120] Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisionsarxiv.org
• Prediction, Explanation, or Over-interpretation?elena-baixy.github.io
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations — LessWronglesswrong.com
• Mapping the Mind of a Large Language Model \ Anthropicanthropic.com