[2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Model

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status Get status notifications via email or slack

[2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Model Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2305.18290 (cs) [Submitted on 29 May 2023 ( v1 ), last revised 29 Jul 2024 (this version, v3)] Title: Direct Preference Optimization: Your Language Model is Secretly a Reward Model Authors: Rafael Rafailov , Archit Sharma , Eric Mitchell , Stefano Ermon , Christopher D. Manning , Chelsea Finn View a PDF of the paper title

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• rlhfbook.com/book.pdfrlhfbook.com
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io
• [2402.14740] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• How RLHF actually works - by Nathan Lambertinterconnects.ai
• Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOarxiv.org
• Nash Learning from Human Feedbackarxiv.org
• Reinforcement learning from human feedback - Wikipediaen.wikipedia.org
• On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationarxiv.org