A Toy Environment For Exploring Reasoning About Reward — LessWrong
tldr: We share a toy environment that we found useful for understanding how reasoning changed over the course of capabilities-focused RL. Over the co…
x A Toy Environment For Exploring Reasoning About Reward — LessWrong AI Frontpage 56 A Toy Environment For Exploring Reasoning About Reward by jenny , Bronson Schoen 25th Mar 2026 AI Alignment Forum 3 min read 7 56 Ω 26 tldr : We share a toy environment that we found useful for understanding how reasoning changed over the course of capabilities-focused RL . Over the course of capabilities-focused RL, the model biases more strongly towards reward hints over direct instruction in this environment. Setup When we noticed the increase in verbalized alignment evaluation awareness during capabilities
saved by
related reading
- Training a Misaligned Reward Seekeralignment.anthropic.com
- Simulated Users & Sad LLMs1a3orn.com
- Reinforcement learning towards broadly and persistently beneficial modelsalignment.openai.com
- Metagaming matters for training, evaluation, and oversightalignment.openai.com
- The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
- Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
- How hard is it to inoculate against misalignment generalization? — LessWronglesswrong.com
- Models Don't "Get Reward" — LessWronglesswrong.com
- Reward is not the optimization target — LessWronglesswrong.com
- Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research — AI Alignment Forumalignmentforum.org
- Natural-emergent-misalignment-from-reward-hacking-paper.pdfassets.anthropic.com
- Alignment Faking Mitigationsalignment.anthropic.com