Modifying LLM Beliefs with Synthetic Document Finetuning

In this post, we study whether we can modify an LLM’s beliefs and investigate whether doing so could decrease risk from advanced AI systems.

Modifying LLM Beliefs with Synthetic Document Finetuning Alignment Science Blog Modifying LLM Beliefs with Synthetic Document Finetuning Rowan Wang April 24, 2025 Avery Griffin ‡ , Johannes Treutlein Ethan Perez, Julian Michael § , Fabien Roger, Sam Marks Anthropic; ‡ MATS; § Scale AI In this post, we study whether we can modify an LLM’s beliefs and investigate whether doing so could decrease risk from advanced AI systems. We describe a pipeline for modifying LLM beliefs via synthetic document finetuning and introduce a suite of evaluations that suggest our pipeline succeeds in inserting all b

saved by

• Stella Jia
• Timothy Kostolansky
• Emil Ryd
• Julian H
• Arjun Khandelwal
• Davis Brown
• Emanuel Ruzak

related reading

• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
• Believe It or Not: How Deeply do LLMs Believe Implanted Facts?alignment.anthropic.com
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment — LessWronglesswrong.com
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Traininggoodfire.ai
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentalignmentpretraining.ai
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org
• [2510.17941] Believe It or Not: How Deeply do LLMs Believe Implanted Facts?arxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• 2308.03958arxiv.org
• Alignment pretraining could backfire — LessWronglesswrong.com