[2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Model

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status Get status notifications via email or slack

[2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Model Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2305.18290 (cs) [Submitted on 29 May 2023 ( v1 ), last revised 29 Jul 2024 (this version, v3)] Title: Direct Preference Optimization: Your Language Model is Secretly a Reward Model Authors: Rafael Rafailov , Archit Sharma , Eric Mitchell , Stefano Ermon , Christopher D. Manning , Chelsea Finn View a PDF of the paper title

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Fine-tune Llama 2 with DPOhuggingface.co
• RLHF | John Lambertjohnwlambert.github.io
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io
• rlhfbook.com/book.pdfrlhfbook.com
• 2401.10020.pdfarxiv.org
• Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencesarxiv.org
• Introduction | RLHF and Post-Training Book by Nathan Lambertrlhfbook.com
• [2402.14740] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsarxiv.org
• Preference Tuning LLMs with Direct Preference Optimization Methodshuggingface.co