[1712.05812] Occam's razor is insufficient to infer the preferences of irrational agents

Abstract:Inverse reinforcement learning (IRL) attempts to infer human rewards or preferences from observed behavior. Since human planning systematically deviates from rationality, several approaches have been tried to account for specific human shortcomings. However, the general problem of inferring the reward function of an agent of unknown rationality has received little attention. Unlike the well-known ambiguity problems in IRL, this one is practically relevant but cannot be resolved by observing the agent's policy in enough environments. This paper shows (1) that a No Free Lunch result implies it is impossible to uniquely decompose a policy into a planning algorithm and reward function, and (2) that even with a reasonable simplicity prior/Occam's razor on the set of decompositions, we cannot distinguish between the true decomposition and others that lead to high regret. To address this, we need simple `normative' assumptions, which cannot be deduced exclusively from observations.

Occam’s razor is insufficient to infer the preferences of irrational agents Stuart Armstrong ∗* † Sören Mindermann* ‡ Future of Humanity Institute Vector Institute arXiv:1712.05812v6 [cs.AI] 11 Jan 2019 University of Oxford University of Toronto…

related reading

• [1712.05812] Occam's razor is insufficient to infer the preferences of irrational agentsarxiv.org
• [1906.09624] On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencearxiv.org
• Reward is not the optimization target — LessWronglesswrong.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Deep Reinforcement Learning from Human Preferencesproceedings.neurips.cc
• RLHF Bookrlhfbook.com
• [2208.10687] The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typesar5iv.labs.arxiv.org
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
• Embedded Agency (full-text version) — LessWronglesswrong.com
• [1711.02827] Inverse Reward Designarxiv.org
• What is AIXI?jan.leike.name