Fitness-Seekers: Generalizing the Reward-Seeking Threat Model

If you think reward-seekers are plausible, you should also think “fitness-seekers” are plausible. But their risks aren’t the same.

The AI safety community often emphasizes reward-seeking as a central case of a misaligned AI alongside scheming (e.g., Cotra’s sycophant vs schemer, Carlsmith’s terminal vs instrumental training-gamer). We are also starting to see signs of reward-seeking-like motivations. But I think insufficient care has gone into delineating this category. If you were to focus on AIs who care about reward in particular[1], you’d be missing some comparably-or-more plausible nearby motivations that make the picture of risk notably more complex. A classic reward-seeker wants high reward on the current…

related reading

• Fitness-Seekers: Generalizing the Reward-Seeking Threat Model — LessWronglesswrong.com
• Risk from fitness-seeking AIs: mechanisms and mitigationssubstack.com
• Risk from fitness-seeking AIs: mechanisms and mitigationsblog.redwoodresearch.org
• The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
• Training a Misaligned Reward Seekeralignment.anthropic.com
• Will reward-seekers respond to distant incentives? — LessWronglesswrong.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Reward is not the optimization target — LessWronglesswrong.com
• New report: “Scheming AIs: Will AIs fake alignment during training in order to get power?”joecarlsmith.com
• Are AIs more likely to pursue on-episode or beyond-episode reward?blog.redwoodresearch.org
• The behavioral selection model for predicting AI motivationsblog.redwoodresearch.org
• Models Don't "Get Reward" — LessWronglesswrong.com