Fail safe(r) at alignment by channeling reward-hacking into a "spillway" motivation

A controlled reward-seeking motivation could make AI safer and more useful

It’s plausible that flawed RL processes will select for misaligned AI motivations.1 Some misaligned motivations are much more dangerous than others. So, developers should plausibly aim to control which kind of misaligned motivations emerge in this case. In particular, we tentatively propose that developers should try to make the most likely generalization of reward hacking a bespoke bundle of benign reward-seeking traits, called a spillway motivation. We call this process spillway design. We think spillway design could have two major benefits: Spillway design might decrease the probability…

saved by

• Kaustubh Kislay
• Arya Pasumarthi

related reading

• Risk from fitness-seeking AIs: mechanisms and mitigationssubstack.com
• The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
• (Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL — LessWronglesswrong.com
• Alignment Faking Mitigationsalignment.anthropic.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Training a Misaligned Reward Seekeralignment.anthropic.com
• Natural-emergent-misalignment-from-reward-hacking-paper.pdfassets.anthropic.com
• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
• How hard is it to inoculate against misalignment generalization? — LessWronglesswrong.com
• Maybe we should pretrain on synthetic data about good-but-reward-hacking AIs — LessWronglesswrong.com
• Reinforcement learning towards broadly and persistently beneficial modelsalignment.openai.com
• Training on Documents about Reward Hacking Induces Reward Hackingalignment.anthropic.com