Reward Function Design: a starter pack — LessWrong

In the companion post “We need a field of Reward Function Design”, I implore researchers to think about what RL reward functions (if any) will lead to RL agents that are not ruthless power-seeking consequentialists. And I further suggested that human social instincts constitutes an intriguing example we should study, since they seem to be an existence proof that such reward functions exist. So what is the general principle of Reward Function Design that underlies the non-ruthless (“ruthful”??) properties of human social instincts? And whatever that general principle is, can we apply it to future RL agent AGIs?

x Reward Function Design: a starter pack — LessWrong Reinforcement learning AI Frontpage 82 Reward Function Design: a starter pack by Steven Byrnes 8th Dec 2025 AI Alignment Forum 3 min read 14 82 Ω 29 In the companion post We need a field of Reward Function Design , I implore researchers to think about what RL reward functions (if any) will lead to RL agents that are not ruthless power-seeking consequentialists. And I further suggested that human social instincts constitutes an intriguing example we should study, since they seem to be an existence proof that such reward functions exist. So wh

related reading

• Reward is not the optimization target — LessWronglesswrong.com
• Reward Is Not Enough — LessWronglesswrong.com
• The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
• Models Don't "Get Reward" — LessWronglesswrong.com
• Why Tool AIs Want to Be Agent AIs · Gwern.netgwern.net
• “Behaviorist” RL reward functions lead to scheming — AI Alignment Forumalignmentforum.org
• Reward Is Not the Optimization Targetturntrout.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• A Toy Environment For Exploring Reasoning About Reward — LessWronglesswrong.com
• Fitness-Seekers: Generalizing the Reward-Seeking Threat Model — LessWronglesswrong.com
• Reward is not the optimization target — AI Alignment Forumalignmentforum.org
• The shard theory of human values — LessWronglesswrong.com