From REINFORCE to Dr. GRPO

A Unified Perspective on LLM Post-Training

From REINFORCE to Dr. GRPO From REINFORCE to Dr. GRPO A Unified Perspective on LLM Post-Training March 27, 2025 Article Self Research Recently, many reinforcement learning (RL) algorithms have been applied to improve the post-training of large language models (LLMs). In this article, we aim to provide a unified perspective on the objectives of these RL algorithms, exploring how they relate to each other through the Policy Gradient Theorem [1] — the fundamental theorem of policy gradient methods. Background Let $\Delta(X)$ be the space of all probability distributions supported over the set $X$

saved by

• Julian H

related reading

• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• RLHF Bookrlhfbook.com
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• Part 3: Intro to Policy Optimization - Spinning Up documentationspinningup.openai.com
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• Harsh Bhatt (@harshbhatt7585) on Xx.com
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• Interactive Visualization of RL Algorithms for LLM Trainingzcy233035.github.io