[2506.11613] Model Organisms for Emergent Misalignment

Abstract:Recent work discovered Emergent Misalignment (EM): fine-tuning large language models on narrowly harmful datasets can lead them to become broadly misaligned. A survey of experts prior to publication revealed this was highly unexpected, demonstrating critical gaps in our understanding of model alignment. In this work, we both advance understanding and provide tools for future research. Using new narrowly misaligned datasets, we create a set of improved model organisms that achieve 99% coherence (vs. 67% prior), work with smaller 0.5B parameter models (vs. 32B), and that induce misalignment using a single rank-1 LoRA adapter. We demonstrate that EM occurs robustly across diverse model sizes, three model families, and numerous training protocols including full supervised fine-tuning. Leveraging these cleaner model organisms, we isolate a mechanistic phase transition and demonstrate that it corresponds to a robust behavioural phase transition in all studied organisms. Aligning large language models is critical for frontier AI safety, yet EM exposes how far we are from achieving this robustly. By distilling clean model organisms that isolate a minimal alignment-compromising change, and where this is learnt, we establish a foundation for future research into understanding and mitigating alignment risks in LLMs.

Model Organisms for Emergent Misalignment Edward Turner * Anna Soligo * Mia Taylor Senthooran Rajamanoharan Neel Nanda Abstract was discovered by Betley et al. (2025b), who termed it ‘emergent misalignment’ (EM). Recent work discovered Emergent Misalignment…

related reading

• Model Organisms for Emergent Misalignment — LessWronglesswrong.com
• Model Organisms for Emergent Misalignmentarxiv.org
• Narrow Misalignment is Hard, Emergent Misalignment is Easy — LessWronglesswrong.com
• [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
• Emergent Misalignment is Easy, Narrow Misalignment is Hardarxiv.org
• Chapter 4: Alignment Science - ARENAlearn.arena.education
• Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs 49 This paper contains model-generated content that might be offensive. 49arxiv.org
• Open problems in emergent misalignment — LessWronglesswrong.com
• Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggersarxiv.org
• We need a better way to evaluate emergent misalignment — LessWronglesswrong.com
• Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research — AI Alignment Forumalignmentforum.org
• Alignment remains a hard, unsolved problem — LessWronglesswrong.com