Recontextualization Mitigates Specification Gaming Without Modifying the Specification — LessWrong

Recontextualization distills good behavior into a context which allows bad behavior. More specifically, recontextualization is a modification to RL which generates completions from prompts that discourage misbehavior, appends those completions to prompts that are more tolerant of misbehavior, and finally reinforces the model on the recontextualized instruction-completion data. Due to the data generation and training prompts differing in their attitude towards misbehavior, recontextualization builds resistance to misbehaviors that the training signal mistakenly reinforces. For example, suppose our reward signal does not robustly penalize deception. Recontextualization generates completions while discouraging deception and then creates training data by updating those completions' prompts to encourage deception. That simple tweak can prevent the model from becoming dishonest! We developed recontextualization concurrently with recent work on inoculation prompting. Wichers et al. and Tan et

x Recontextualization Mitigates Specification Gaming Without Modifying the Specification — LessWrong MATS Program Prompt Engineering Reward Functions AI Frontpage 2025 Top Fifty: 13 % 144 Recontextualization Mitigates Specification Gaming Without Modifying the Specification by ariana_azarbal , Victor Gillioz , TurnTrout , cloud 14th Oct 2025 AI Alignment Forum 13 min read 15 144 Ω 57 Recontextualization distills good behavior into a context which allows bad behavior. More specifically, recontextualization is a modification to RL which generates completions from prompts that discourage misbehav

related reading

• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Natural-emergent-misalignment-from-reward-hacking-paper.pdfassets.anthropic.com
• Training a Reward Hacker Despite Perfect Labels — LessWronglesswrong.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• A Toy Environment For Exploring Reasoning About Reward — LessWronglesswrong.com
• Paper: Prompt Optimization Makes Misalignment Legible — LessWronglesswrong.com
• Specification gaming: the flip side of AI ingenuity — Google DeepMinddeepmind.google
• (Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL — LessWronglesswrong.com
• Should We Train Against (CoT) Monitors? — LessWronglesswrong.com
• Features as Rewards: Using Interpretability to Reduce Hallucinationsgoodfire.ai
• Reward hacking behavior can generalize across tasks — AI Alignment Forumalignmentforum.org
• Models Don't "Get Reward" — LessWronglesswrong.com