Generalisation Hacking: a first look at adversarial generalisation failures in deliberative alignment — LessWrong
Background Deliberative alignment is a powerful post-training alignment technique that involves generating and training on re-contextualised supervis…
x Generalisation Hacking: a first look at adversarial generalisation failures in deliberative alignment — LessWrong AI Frontpage 54 Generalisation Hacking: a first look at adversarial generalisation failures in deliberative alignment by Cam , Puria 17th Nov 2025 AI Alignment Forum 10 min read 2 54 Ω 16 Background Deliberative alignment is a powerful post-training alignment technique that involves generating and training on re-contextualised supervised fine-tuning (SFT) datasets generated with a set of principles in context. The process takes three steps: With the set of principles [1] (hencefo
related reading
- Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
- Alignment Faking Mitigationsalignment.anthropic.com
- Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
- Natural-emergent-misalignment-from-reward-hacking-paper.pdfassets.anthropic.com
- How far does alignment midtraining generalize?alignment.openai.com
- How hard is it to inoculate against misalignment generalization? — LessWronglesswrong.com
- Training a Misaligned Reward Seekeralignment.anthropic.com
- Deep Deceptiveness — LessWronglesswrong.com
- Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment — LessWronglesswrong.com
- Training on Documents about Reward Hacking Induces Reward Hackingalignment.anthropic.com
- Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentalignmentpretraining.ai
- [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org