Reinforcement Learning | RLHF and Post-Training Book by Nathan Lambert

Policy gradient methods for RLHF and LLM post-training, including PPO, REINFORCE, RLOO, GRPO, and implementation details.

--> RLHF Book --> Reinforcement Learning | RLHF and Post-Training Book by Nathan Lambert Reinforcement Learning from Human Feedback A short introduction to RLHF and post-training focused on language models. Nathan Lambert Lecture 3: Understanding Policy Gradient Algorithms for RL on LLMs Lecture 4: Implementing RL Algorithms for LLMs Reinforcement Learning In the RLHF process, the reinforcement learning algorithm slowly updates the model’s weights with respect to feedback from a reward model. The policy – the model being trained – generates completions to prompts in the training set, then the

saved by

• Timothy Kostolansky

related reading

• Why GRPO is Important and How it Worksghost.oxen.ai
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• From REINFORCE to Dr. GRPOlancelqf.github.io
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• Lightweight Guide to understanding GRPO and RL principles - Musings of Muraligitlostmurali.com
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com