Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? Paul Gölz Cornell University paulgoelz@cornell.edu Nika Haghtalab UC Berkeley nika@berkeley.edu Kunhe Yang UC Berkeley kunheyang@berkeley.edu Abstract After pre-training, large language models are aligned with human preferences based on pairwise comparisons. State-of-the-art alignment methods (such as PPO-based RLHF and DPO) are built on the assumption of aligning with a single preference model, despite being deployed in settings

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationarxiv.org
• [2503.10990] Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumarxiv.org
• [2605.11134] Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingarxiv.org
• Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelarxiv.org
• RLHF | John Lambertjohnwlambert.github.io
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studyarxiv.org
• Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOarxiv.org
• Scaling Laws for Reward Model Overoptimizationarxiv.org
• A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicityarxiv.org
• Metric Distortion Under Probabilistic Votingarxiv.org
• Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencesarxiv.org