RLHF | John Lambert

Reinforcement Learning from Human Feedback

Table of Contents: Overview Reward Model Training PPO Direct Preference Optimization (DPO) DPO: Deriving the Optimum of the KL-Constrained Reward Maximization Objective DPO: Solving for Reward, Using the Optimal Policy DPO: Revealing the DPO Loss Function DPO: Deriving the DPO Objective Under the Bradley-Terry Model GRPO Post-Training in the Post-RLHF Era Overview Reinforcement learning from human feedback (RLHF) align models (and language modeling objective) with users’ values to be truthful, non-toxic, and helpful to the user, through the use of a trained reward model. Because…

saved by

• Jackson Mowatt Gok
• Akira Yoshiyama

related reading

• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Fine-tune Llama 2 with DPOhuggingface.co
• rlhfbook.com/book.pdfrlhfbook.com
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• Reinforcement learning from human feedback - Wikipediaen.wikipedia.org
• Rethinking the Role of PPO in RLHF – The Berkeley Artificial Intelligence Research Blogbair.berkeley.edu