Measuring Reward-Seeking by Instilling Contrastive Beliefs

We developed Contrastive Synthetic Document Finetuning (Contrastive SDF), a new test for whether an AI model changes its behavior when it has different beliefs about what a grader rewards.

Measuring Reward-Seeking by Instilling Contrastive Beliefs ← Back to OpenAI Alignment Blog Measuring Reward-Seeking by Instilling Contrastive Beliefs Jul 21, 2026 · Axel Højmark (Apollo Research), Jérémy Scheurer (Apollo Research), Jenny Nitishinskaya, Felix Hofstätter (Apollo Research), Jason Wolfe, Theodore Ehrenborg (Apollo Research), Bronson Schoen (Apollo Research), Alexander Meinke (Apollo Research) Correspondence: jenny [at] openai.com Read the paper In Brief We developed a new test, Contrastive Synthetic Document Finetuning (Contrastive SDF), for whether an AI model would change i

saved by

• Emil Ryd
• Julian H
• Ishaan Panigrahi
• Goutham N

related reading

• [2607.18966] Measuring Reward-Seeking via Contrastive Belief Updatesarxiv.org
• Reinforcement learning towards broadly and persistently beneficial modelsalignment.openai.com
• Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
• Training on Documents about Reward Hacking Induces Reward Hackingalignment.anthropic.com
• [2606.12016] Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalizationarxiv.org
• Studying metagaming latents in language modelsalignment.openai.com
• Why We Are Excited About Confessionsalignment.openai.com
• Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivesarxiv.org
• [2606.26071] Model Forensics: Investigating Whether Concerning Behavior Reflects Misalignmentarxiv.org
• Believe It or Not: How Deeply do LLMs Believe Implanted Facts?alignment.anthropic.com
• Training a Misaligned Reward Seekeralignment.anthropic.com
• How far does alignment midtraining generalize?alignment.openai.com