[1711.02827] Inverse Reward Design

Autonomous agents optimize the reward function we give them. What they don't know is how hard it is for us to design a reward function that actually captures what we want. When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios. Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior. Our insight is that reward functions are merely observations about what the designer actually wants, and that they should be interpreted in the context in which they were designed. We introduce inverse reward design (IRD) as the problem of inferring the true objective based on the designed reward and the training MDP. We introduce approximate methods for solving IRD problems, and use their solution to plan risk-averse behavior in test MDPs. Empirical results suggest that this approach can help alleviate negative side effects of misspecified reward functions and mitigate reward hacking.

Inverse Reward Design Dylan Hadfield-Menell Smitha Milli Pieter Abbeel∗ Stuart Russell Anca D. Dragan Department of Electrical Engineering and Computer Science University of California, Berkeley Berkeley, CA 94709 {dhm, smilli, pabbeel, russell, anca}@cs.berkeley.edu…

related reading

• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Reward is not the optimization target — LessWronglesswrong.com
• Reward Is Not Enough — LessWronglesswrong.com
• Reward Function Design: a starter pack — LessWronglesswrong.com
• Reward Is Not the Optimization Targetturntrout.com
• Models Don't "Get Reward" — LessWronglesswrong.com
• [1906.09624] On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencearxiv.org
• Specification gaming: the flip side of AI ingenuity — Google DeepMinddeepmind.google
• [1712.05812] Occam's razor is insufficient to infer the preferences of irrational agentsarxiv.org
• Training a Misaligned Reward Seekeralignment.anthropic.com
• [2201.03544] The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelsarxiv.org
• Reward is not the optimization target — AI Alignment Forumalignmentforum.org