[2508.00161] Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs

Abstract:The releases of powerful open-weight large language models (LLMs) are often not accompanied by access to their full training data. Existing interpretability methods, particularly those based on activations, often require or assume distributionally similar data. This is a significant limitation when detecting and defending against novel potential threats like backdoors, which are by definition out-of-distribution. In this work, we introduce a new method for understanding, monitoring and controlling fine-tuned LLMs that interprets weights, rather than activations, thereby sidestepping the need for data that is distributionally similar to the unknown training data. We demonstrate that the top singular vectors of the weight difference between a fine-tuned model and its base model correspond to newly acquired behaviors. By monitoring the cosine similarity of activations along these directions, we can detect salient behaviors introduced during fine-tuning with high precision. For backdoored models that bypass safety mechanisms when a secret trigger is present, our method stops up to 100% of attacks with a false positive rate below 1%. For models that have undergone unlearning, we detect inference on erased topics with accuracy up to 95.42% and can even steer the model to recover "unlearned" information. Besides monitoring, our method also shows potential for pre-deployment model auditing: by analyzing commercial instruction-tuned models (OLMo, Llama, Qwen), we are able to uncover model-specific fine-tuning focus including mathematical problem solving, emoji usage, and Midjourney prompt generation.

[2508.00161] Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2508.00161 (cs) [Submitted on 31 Jul 2025 ( v1 ), last revised 21 Apr 2026 (this version, v3)] Title: Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs Authors: Ziqian Zhong , Aditi Raghunathan View a PDF of the paper titled Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs, by Z

related reading

• [2508.00161] Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMsarxiv.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• Transformer Circuits Threadtransformer-circuits.pub
• Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• You Can Catch Sleeper Agents by Teaching Another Model to Imitate Them — LessWronglesswrong.com
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• [2312.06942] AI Control: Improving Safety Despite Intentional Subversionarxiv.org
• Mapping the Mind of a Large Language Model \ Anthropicanthropic.com
• Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Traininggoodfire.ai
• [2401.05566] Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingarxiv.org
• Covert Malicious Finetuning — LessWronglesswrong.com
• [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org