Studying metagaming latents in language models

Reasoning about how a task will be monitored or rewarded appears to draw on several overlapping processes, not a single mechanism.

Reasoning about how a task will be monitored or rewarded appears to draw on several overlapping processes, not a single mechanism. Oct 6, 2026 · Xiaojun Xu, Jenny Nitishinskaya, Bronson Schoen (Apollo Research), Dan Mossing (work done while at OpenAI), Tom Dupre la Tour Summary We study what happens inside an AI model when it starts metagaming, or reasoning about how a task is being evaluated or rewarded instead of simply attempting the task. By examining a capabilities-focused OpenAI o3 reinforcement learning run, we identified internal patterns associated with different kinds of…

saved by

• Ishaan Panigrahi

related reading

• Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivesarxiv.org
• Metagaming matters for training, evaluation, and oversightalignment.openai.com
• Measuring Reward-Seeking by Instilling Contrastive Beliefsalignment.openai.com
• Training a Misaligned Reward Seekeralignment.anthropic.com
• [2607.18966] Measuring Reward-Seeking via Contrastive Belief Updatesarxiv.org
• Auditing language models for hidden objectives — LessWronglesswrong.com
• Training on Documents about Reward Hacking Induces Reward Hackingalignment.anthropic.com
• Natural-emergent-misalignment-from-reward-hacking-paper.pdfassets.anthropic.com
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationarxiv.org
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• Externalized reasoning oversight: a research direction for language model alignment — AI Alignment Forumalignmentforum.org
• Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org