Reinforcement Learning | RLHF and Post-Training Book by Nathan Lambert

Policy gradient methods for RLHF and LLM post-training, including PPO, REINFORCE, RLOO, GRPO, and implementation details.

--> RLHF Book --> Reinforcement Learning | RLHF and Post-Training Book by Nathan Lambert Reinforcement Learning from Human Feedback A short introduction to RLHF and post-training focused on language models. Nathan Lambert Lecture 3: Understanding Policy Gradient Algorithms for RL on LLMs Lecture 4: Implementing RL Algorithms for LLMs Reinforcement Learning In the RLHF process, the reinforcement learning algorithm slowly updates the model’s weights with respect to feedback from a reward model. The policy – the model being trained – generates completions to prompts in the training set, then the

saved by

• Timothy Kostolansky

related reading

• Why GRPO is Important and How it Worksghost.oxen.ai
• RLHF | John Lambertjohnwlambert.github.io
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• State of RL for reasoning LLMs | A. Weersaweers.de
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• rlhfbook.com/book.pdfrlhfbook.com
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• RLHF Bookrlhfbook.com