A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyond

A Blog post by Karina Zadorozhny on Hugging Face

A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyond Back to Articles a]:hidden"> A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyond Community Article Published January 19, 2026 Upvote 29 +23 Karina Zadorozhny karina-zadorozhny Follow Definitions Let's define standard reinforcement learning terms with an LLM setup in mind. State s t s_t s t ​ : The current context which is the original user prompt and all tokens generated so far Example: Prompt: "The sky is..." → \rightarrow → State: ["The", "sky", "is"] in the token-space Actio

saved by

• Jeff Brown

related reading

• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• rlhfbook.com/book.pdfrlhfbook.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• RLHF Bookrlhfbook.com
• From REINFORCE to Dr. GRPOlancelqf.github.io
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• Rethinking the Role of PPO in RLHF – The Berkeley Artificial Intelligence Research Blogbair.berkeley.edu
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studyarxiv.org