[2510.05092] Learning to Interpret Weight Differences in Language Models

Abstract:Finetuning (pretrained) language models is a standard approach for updating their internal parametric knowledge and specializing them to new tasks and domains. However, the corresponding model weight changes ("weight diffs") are not generally interpretable. While inspecting the finetuning dataset can give a sense of how the model might have changed, these datasets are often not publicly available or are too large to work with directly. Towards the goal of comprehensively understanding weight diffs in natural language, we introduce Diff Interpretation Tuning (DIT), a method that trains models to describe their own finetuning-induced modifications. Our approach uses synthetic, labeled weight diffs to train a DIT-adapter, which can be applied to a compatible finetuned model to make it describe how it has changed. We demonstrate in two proof-of-concept settings (reporting hidden behaviors and summarizing finetuned knowledge) that our method enables models to describe their finetuning-induced modifications using accurate natural language descriptions.

[2510.05092] Learning to Interpret Weight Differences in Language Models Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2510.05092 (cs) [Submitted on 6 Oct 2025 ( v1 ), last revised 21 Mar 2026 (this version, v4)] Title: Learning to Interpret Weight Differences in Language Models Authors: Avichal Goel , Yoon Kim , Nir Shavit , Tony T. Wang View a PDF of the paper titled Learning to Interpret Weight Differences in Language Models, by Avichal Goel and 3 other

related reading

• [2510.05092] Learning to Interpret Weight Differences in Language Modelsarxiv.org
• Transformer Circuits Threadtransformer-circuits.pub
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• Language models can explain neurons in language modelsopenaipublic.blob.core.windows.net
• Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
• Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences — LessWronglesswrong.com
• Inkling: Our Open-Weights Model - Thinking Machines Labthinkingmachines.ai
• Weight-Sparse Circuits May Be Interpretable Yet Unfaithful — LessWronglesswrong.com
• [2508.00161] Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMsarxiv.org
• [2508.00161] Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMsarxiv.org
• Recent Advances in Language Model Fine-tuningruder.io
• Prism: mapping interpretable concepts and features in a latent space of language | thesephist.comthesephist.com