Discovering Undesired Rare Behaviors via Model Diff Amplification
Model diff amplification (also known as logit diff amplification, or LDA) is a simple method for efficiently identifying rare, unexpected effects of a training run on model behavior. It's useful tool for red-teaming and evaluation, monitoring training runs, detecting emergent misalignment, and detecting backdoors/sleeper agents.
Update (November 2025): Since this post was published, the method it describes has also come to be called logit diff amplification (LDA). One of the biggest issues with LLMs is that training can cause unexpected and undesired behaviors to show up in specific circumstances. Because they only appear occasionally, finding them before deploying a model is a needle-in-a-haystack problem - they often aren't caught by standard evaluations, and end up being found after deployment by bewildered users or delighted jailbreakers. This issue appears in striking incidents like ChatGPT encouraging…
saved by
related reading
- Narrow Misalignment is Hard, Emergent Misalignment is Easy — LessWronglesswrong.com
- [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
- Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
- Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
- A “diff” tool for AI: Finding behavioral differences in new models \ Anthropicanthropic.com
- [2401.05566] Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingarxiv.org
- You Can Catch Sleeper Agents by Teaching Another Model to Imitate Them — LessWronglesswrong.com
- Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org
- [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Traininggoodfire.ai
- Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
- Refusal in LLMs is mediated by a single direction — LessWronglesswrong.com