Fine-tune Llama 2 with DPO

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Introduction Reinforcement Learning from Human Feedback (RLHF) has become the de facto last training step of LLMs such as GPT-4 or Claude to ensure that the language model's outputs are aligned with human expectations such as chattiness or safety features. However, it brings some of the complexity of RL into NLP: we need to build a good reward function, train the model to estimate the value of a state, and at the same time be careful not to strive too far from the original model and produce gibberish instead of sensible text. Such a process is quite involved requiring a number of complex…

saved by

• Saurav Kumar
• Tiết Nguyên Khôi Nguyễn
• Maisha Thasin

related reading

• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• RLHF | John Lambertjohnwlambert.github.io
• Preference Tuning LLMs with Direct Preference Optimization Methodshuggingface.co
• State of RL for reasoning LLMs | A. Weersaweers.de
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studyarxiv.org
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• Direct Preference Optimization with Dynamic $\beta$neurips.cc
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• Fine-Tuning Llama-2: Tailoring Models to Unique Applicationsanyscale.com
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co