Reward Is Not the Optimization Target

RL doesn't train reward optimizers. Reward chisels cognition into agents. Worry less about safe objectives, more about good cognition.

Table of Contents Reward probably won’t be a deep RL agent’s primary optimization target The siren-like suggestiveness of the word “reward” When is reward the optimization target of the agent? Anticipated questions Dropping the old hypothesis Implications Citation Similar posts Appendix: The field of RL thinks reward is the optimization target Footnotes In this essay, I call an agent a “reward optimizer” if it not only gets lots of reward, but if it reliably makes choices like “reward but no task completion” (e.g. receiving reward without eating pizza) over “task completion but no reward” (e.g

saved by

• Tazik Sh
• Timothy Kostolansky
• Julian H

related reading

• Reward is not the optimization target — LessWronglesswrong.com
• Reward is not the optimization target — AI Alignment Forumalignmentforum.org
• Models Don't "Get Reward" — LessWronglesswrong.com
• Why Tool AIs Want to Be Agent AIs · Gwern.netgwern.net
• Reinforcement Learning for Knowledge Awareness – kalomaze's kalomazing blogkalomaze.bearblog.dev
• The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• Reward Is Not Enough — LessWronglesswrong.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Fitness-Seekers: Generalizing the Reward-Seeking Threat Model — LessWronglesswrong.com