A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyond
huggingface.co · 7,753 words · saved by 1 readers
A Blog post by Karina Zadorozhny on Hugging Face
A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyond Back to Articles a]:hidden"> A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyond Community Article Published January 19, 2026 Upvote 29 +23 Karina Zadorozhny karina-zadorozhny Follow Definitions Let's define standard reinforcement learning terms with an LLM setup in mind. State s t s_t s t : The current context which is the original user prompt and all tokens generated so far Example: Prompt: "The sky is..." → \rightarrow → State: ["The", "sky", "is"] in the token-space Actio
saved by
related reading
- The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
- rlhfbook.com/book.pdfrlhfbook.com
- A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
- PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
- From REINFORCE to Dr. GRPOlancelqf.github.io
- State of RL for reasoning LLMs | A. Weersaweers.de
- RLHF Bookrlhfbook.com
- RLHF | John Lambertjohnwlambert.github.io
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- Harsh Bhatt (@harshbhatt7585) on Xx.com
- A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
- GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com