[1712.05812] Occam's razor is insufficient to infer the preferences of irrational agents

Abstract:Inverse reinforcement learning (IRL) attempts to infer human rewards or preferences from observed behavior. Since human planning systematically deviates from rationality, several approaches have been tried to account for specific human shortcomings. However, the general problem of inferring the reward function of an agent of unknown rationality has received little attention. Unlike the well-known ambiguity problems in IRL, this one is practically relevant but cannot be resolved by observing the agent's policy in enough environments. This paper shows (1) that a No Free Lunch result implies it is impossible to uniquely decompose a policy into a planning algorithm and reward function, and (2) that even with a reasonable simplicity prior/Occam's razor on the set of decompositions, we cannot distinguish between the true decomposition and others that lead to high regret. To address this, we need simple `normative' assumptions, which cannot be deduced exclusively from observations.

[1712.05812] Occam's razor is insufficient to infer the preferences of irrational agents Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Artificial Intelligence arXiv:1712.05812 (cs) [Submitted on 15 Dec 2017 ( v1 ), last revised 11 Jan 2019 (this version, v6)] Title: Occam's razor is insufficient to infer the preferences of irrational agents Authors: Stuart Armstrong , Sören Mindermann View a PDF of the paper titled Occam's razor is insufficient to infer the preferences of irratio

related reading

• [1712.05812] Occam's razor is insufficient to infer the preferences of irrational agentsarxiv.org
• [1906.09624] On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencearxiv.org
• Reward is not the optimization target — LessWronglesswrong.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• RLHF Bookrlhfbook.com
• Deep Reinforcement Learning from Human Preferencesproceedings.neurips.cc
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• [1912.01683] Optimal Policies Tend to Seek Powerarxiv.org
• [2208.10687] The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typesar5iv.labs.arxiv.org
• Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
• [AN #70]: Agents that help humans who are still learning about their own preferences — LessWronglesswrong.com
• Irrationality as a Defense Mechanism for Reward-hacking — LessWronglesswrong.com