[2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Model

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status Get status notifications via email or slack

[2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Model Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2305.18290 (cs) [Submitted on 29 May 2023 ( v1 ), last revised 29 Jul 2024 (this version, v3)] Title: Direct Preference Optimization: Your Language Model is Secretly a Reward Model Authors: Rafael Rafailov , Archit Sharma , Eric Mitchell , Stefano Ermon , Christopher D. Manning , Chelsea Finn View a PDF of the paper title

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• [2406.11717] Refusal in Language Models Is Mediated by a Single Directionarxiv.org
• [2309.00267] RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackarxiv.org
• [2507.06187] The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainsarxiv.org
• [2402.14740] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsarxiv.org
• LLM Post-Training: A Deep Dive into Reasoning Large Language Modelsarxiv.org
• [2503.10965] Auditing language models for hidden objectivesarxiv.org
• 2401.10020.pdfarxiv.org
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io
• Fine-tune Llama 2 with DPOhuggingface.co
• [2309.01809] Are Emergent Abilities in Large Language Models just In-Context Learning?arxiv.org
• [2302.08582] Pretraining Language Models with Human Preferencesarxiv.org