Fine-tune Llama 2 with DPO

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Introduction Reinforcement Learning from Human Feedback (RLHF) has become the de facto last training step of LLMs such as GPT-4 or Claude to ensure that the language model's outputs are aligned with human expectations such as chattiness or safety features. However, it brings some of the complexity of RL into NLP: we need to build a good reward function, train the model to estimate the value of a state, and at the same time be careful not to strive too far from the original model and produce gibberish instead of sensible text. Such a process is quite involved requiring a number of complex…

saved by

• Saurav Kumar
• Tiết Nguyên Khôi Nguyễn
• Maisha Thasin

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• RLHF | John Lambertjohnwlambert.github.io
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• LLM Training: RLHF and Its Alternativesmagazine.sebastianraschka.com
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studyarxiv.org
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• Preference Tuning LLMs with Direct Preference Optimization Methodshuggingface.co
• rlhfbook.com/book.pdfrlhfbook.com
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• [2402.14740] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsarxiv.org