Reward is not the optimization target — AI Alignment Forum
This insight was made possible by many conversations with Quintin Pope, where he challenged my implicit assumptions about alignment. I’m not sure who…
x Reward is not the optimization target — AI Alignment Forum Best of LessWrong 2022 Reinforcement learning Inner Alignment Reward Functions Wireheading Shard Theory Outer Alignment Deconfusion AI Frontpage 94 Reward is not the optimization target by TurnTrout 25th Jul 2022 12 min read 128 94 This insight was made possible by many conversations with Quintin Pope, where he challenged my implicit assumptions about alignment. I’m not sure who came up with this particular idea. In this essay, I call an agent a “reward optimizer” if it not only gets lots of reward, but if it reliably makes choices l
related reading
- Reward is not the optimization target — LessWronglesswrong.com
- Reward Is Not the Optimization Targetturntrout.com
- Models Don't "Get Reward" — LessWronglesswrong.com
- The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
- Why Tool AIs Want to Be Agent AIs · Gwern.netgwern.net
- Reward Is Not Enough — LessWronglesswrong.com
- Risks from Learned Optimization: Introduction — LessWronglesswrong.com
- Reward Function Design: a starter pack — LessWronglesswrong.com
- Fitness-Seekers: Generalizing the Reward-Seeking Threat Model — LessWronglesswrong.com
- Arguments against myopic training — LessWronglesswrong.com
- A Toy Environment For Exploring Reasoning About Reward — LessWronglesswrong.com
- why assume AGIs will optimize for fixed goals? — LessWronglesswrong.com