[2208.10687] The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types

When inferring reward functions from human behavior (be it demonstrations, comparisons, physical corrections, or e-stops), it has proven useful to model the human as making noisy-rational choices, with a “rationality c…

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types Gaurav R. Ghosal 1 \equalcontrib , Matthew Zurek 2 \equalcontrib , Daniel S. Brown 3 , Anca D. Dragan 1 Abstract When inferring reward functions from human behavior (be it demonstrations, comparisons, physical corrections, or e-stops), it has proven useful to model the human as making noisy-rational choices, with a “rationality coefficient” capturing how much noise or entropy we expect to see in the human behavior. Prior work typically sets the rationality level to a constant value, regardless of th

related reading

• Deep Reinforcement Learning from Human Preferencesproceedings.neurips.cc
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Scaling Laws for Reward Model Overoptimizationarxiv.org
• [2607.18966] Measuring Reward-Seeking via Contrastive Belief Updatesarxiv.org
• Data-driven equation discovery reveals nonlinear reinforcement learning in humanspmc.ncbi.nlm.nih.gov
• [2602.11399] Can We Really Learn One Representation to Optimize All Rewards?arxiv.org
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• Models Don't "Get Reward" — LessWronglesswrong.com
• RLHF | John Lambertjohnwlambert.github.io
• Trajectory Improvement and Reward Learning from Comparative Language Feedbackarxiv.org
• [2309.00267] RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackarxiv.org
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io