Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO Ruizhe Shi Tsinghua University srz21@mails.tsinghua.edu.cn &Minhak Song ∗ * KAIST minhaksong@kaist.ac.kr Runlong Zhou University of Washington vectorzh@cs.washington.edu &Zihan Zhang University of Washington zihanz46@uw.edu Maryam Fazel University of Washington mfazel@uw.edu & Simon S. Du University of Washington ssdu@cs.washington.edu Equal contribution.Work done while Minhak Song was visiting the University of Washington. Abstract We present a fine-grained theoretical analysis of the performance gap between

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• RLHF | John Lambertjohnwlambert.github.io
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studyarxiv.org
• Deep Reinforcement Learning from Human Preferencesproceedings.neurips.cc
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• [2602.11399] Can We Really Learn One Representation to Optimize All Rewards?arxiv.org
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org
• Scaling Laws for Reward Model Overoptimizationarxiv.org
• Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?arxiv.org