[2505.17120] Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions
Abstract:We have only limited understanding of how and why large language models (LLMs) respond in the ways that they do. Their neural networks have proven challenging to interpret, and we are only beginning to tease out the function of individual neurons and circuits within them. However, another path to understanding these systems is to investigate and develop their capacity to explain their own functioning. Here, we show that i) LLMs can accurately describe quantitative features of their own internal processes during certain kinds of decision-making and ii) that it is possible to improve these capabilities through training. To do so, we fine-tuned GPT-4o and GPT-4o-mini to make decisions in a wide variety of complex contexts (e.g., choosing between condos, loans, vacations, etc.) according to randomly-generated, quantitative preferences about how to weigh different attributes (e.g., the relative importance of natural light versus quiet surroundings for condos). We demonstrate that the LLMs can accurately report these preferences (i.e., the weights that they learned to give to different attributes during decision-making). Next, we demonstrate that these LLMs can be fine-tuned to explain their decision-making even more accurately. Finally, we demonstrate that this training generalizes: It improves the ability of the models to accurately explain how they make other complex decisions, not just decisions they have been fine-tuned to make. This work is a step towards training LLMs to accurately and broadly report on their own internal processes -- a possibility that would yield substantial benefits for interpretability, control, and safety.
Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisions Dillon Plunkett Adam Morris Northeastern University Princeton University plunkett@g.harvard.edu thatadammorris@gmail.com arXiv:2505.17120v2 [cs.CL] 10 Nov 2025…
saved by
related reading
- [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
- [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
- Training Language Models to Explain Their Own Computationsarxiv.org
- [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
- Training Language Models to Explain Their Own Computationsarxiv.org
- Emergent introspective awareness in large language models \ Anthropicanthropic.com
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- [2508.14802] Privileged Self-Access Matters for Introspection in AIarxiv.org
- [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org
- Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
- [2410.13787] Looking Inward: Language Models Can Learn About Themselves by Introspectionarxiv.org
- [2501.11120] Tell me about yourself: LLMs are aware of their learned behaviorsarxiv.org