Modifying LLM Beliefs with Synthetic Document Finetuning

In this post, we study whether we can modify an LLM’s beliefs and investigate whether doing so could decrease risk from advanced AI systems.

Modifying LLM Beliefs with Synthetic Document Finetuning Alignment Science Blog Modifying LLM Beliefs with Synthetic Document Finetuning Rowan Wang April 24, 2025 Avery Griffin ‡ , Johannes Treutlein Ethan Perez, Julian Michael § , Fabien Roger, Sam Marks Anthropic; ‡ MATS; § Scale AI In this post, we study whether we can modify an LLM’s beliefs and investigate whether doing so could decrease risk from advanced AI systems. We describe a pipeline for modifying LLM beliefs via synthetic document finetuning and introduce a suite of evaluations that suggest our pipeline succeeds in inserting all b

saved by

• Stella Jia
• Timothy Kostolansky
• Emil Ryd
• Julian H
• Davis Brown
• Emanuel Ruzak

related reading

• Believe It or Not: How Deeply do LLMs Believe Implanted Facts?alignment.anthropic.com
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• Unfamiliar Finetuning Examples Control How Language Models Hallucinatearxiv.org
• [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
• How confessions can keep language models honest | OpenAIopenai.com
• Alignment faking in large language modelsarxiv.org
• [2506.17209] Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencyarxiv.org
• Gemini 3 is Evaluation-Paranoid and Contaminated — LessWronglesswrong.com
• Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs 49 This paper contains model-generated content that might be offensive. 49arxiv.org
• [2510.04340] Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timearxiv.org