Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

N/A

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences Corby Rosset∗ Ching-An Cheng Arindam Mitra Michael Santacroce Ahmed Awadallah∗ Tengyang Xie∗ Microsoft Research arXiv:2404.03715v1 [cs.LG] 4 Apr 2024…

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• Nash Learning from Human Feedbackarxiv.org
• [2503.10990] Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumarxiv.org
• 2401.10020.pdfarxiv.org
• RLHF | John Lambertjohnwlambert.github.io
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
• Fine-tune Llama 2 with DPOhuggingface.co
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io
• On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationarxiv.org