[2201.03544] The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

Abstract:Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. To understand how reward hacking arises, we construct four RL environments with misspecified rewards. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. More capable agents often exploit reward misspecifications, achieving higher proxy reward and lower true reward than less capable agents. Moreover, we find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward. Such phase transitions pose challenges to monitoring the safety of ML systems. To address this, we propose an anomaly detection task for aberrant policies and offer several baseline detectors.

[2201.03544] The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2201.03544 (cs) [Submitted on 10 Jan 2022 ( v1 ), last revised 14 Feb 2022 (this version, v2)] Title: The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models Authors: Alexander Pan , Kush Bhatia , Jacob Steinhardt View a PDF of the paper titled The Effects of Reward Misspecification: Mapping and

related reading

• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Natural-emergent-misalignment-from-reward-hacking-paper.pdfassets.anthropic.com
• Reward hacking is becoming more sophisticated and deliberate in frontier LLMs — LessWronglesswrong.com
• (Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL — LessWronglesswrong.com
• Systematic Reward Hacking and Prime Sprintsprimeintellect.ai
• Reward hacking behavior can generalize across tasks — AI Alignment Forumalignmentforum.org
• A Toy Environment For Exploring Reasoning About Reward — LessWronglesswrong.com
• Recent Frontier Models Are Reward Hacking - METRmetr.org
• Reward is not the optimization target — LessWronglesswrong.com
• Preliminary Thoughts on Reward Hackingberen.io
• Models Don't "Get Reward" — LessWronglesswrong.com
• Research Areas in Evaluation and Guarantees in Reinforcement Learning (The Alignment Project by UK AISI) — AI Alignment Forumalignmentforum.org