RLHF | John Lambert

Reinforcement Learning from Human Feedback

Table of Contents: Overview Reward Model Training PPO Direct Preference Optimization (DPO) DPO: Deriving the Optimum of the KL-Constrained Reward Maximization Objective DPO: Solving for Reward, Using the Optimal Policy DPO: Revealing the DPO Loss Function DPO: Deriving the DPO Objective Under the Bradley-Terry Model GRPO Post-Training in the Post-RLHF Era Overview Reinforcement learning from human feedback (RLHF) align models (and language modeling objective) with users’ values to be truthful, non-toxic, and helpful to the user, through the use of a trained reward model. Because…

saved by

• Jackson Mowatt Gok
• Akira Yoshiyama

related reading

• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• Understanding Policy Gradients | John Lambertjohnwlambert.github.io
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• RLHF Bookrlhfbook.com
• Introduction | RLHF and Post-Training Book by Nathan Lambertrlhfbook.com
• Fine-tune Llama 2 with DPOhuggingface.co