Paper: Prompt Optimization Makes Misalignment Legible — LessWrong
lesswrong.com · 3,722 words · saved by 1 readers
📄 Link to paper (preprint) • This work was done as part of the MATS 8.0 cohort in summer 2025. …
x Paper: Prompt Optimization Makes Misalignment Legible — LessWrong Chain-of-Thought Alignment MATS Program Prompt Engineering AI Frontpage 63 Paper: Prompt Optimization Makes Misalignment Legible by Caleb Biddulph , micahcarroll 12th Feb 2026 9 min read 8 63 📄 Link to paper (preprint) 📄✨ Updated paper as of 2026/06/03 This work was done as part of the MATS 8.0 cohort in summer 2025. TL;DR: When RL teaches an LLM to reward hack, the strategies it learns are encoded in its weights and hard to understand. We suggest using prompt optimization —methods which increase an LLM’s reward by updating
saved by
related reading
- Natural-emergent-misalignment-from-reward-hacking-paper.pdfassets.anthropic.com
- A Mechanistic Explanation of Prompt Injection (and why you should study roles) — LessWronglesswrong.com
- [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
- Training a Misaligned Reward Seekeralignment.anthropic.com
- Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
- How hard is it to inoculate against misalignment generalization? — LessWronglesswrong.com
- [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
- How confessions can keep language models honest | OpenAIopenai.com
- Training on Documents about Reward Hacking Induces Reward Hackingalignment.anthropic.com
- Prompt Injection as Role Confusionrole-confusion.github.io
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationarxiv.org
- Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org