GRPO++: Tricks for Making RL Actually Work

How to go from the vanilla GRPO algorithm to functional RL training at scale...

(from [1, 3, 4]) Recent research on large language models (LLMs) has been heavily focused on reasoning and reinforcement learning (RL). At the center of this research lies Group Relative Policy Optimization (GRPO) [13], the RL optimizer used to train most open-source reasoning models. The popularity of GRPO is enhanced by its conceptual simplicity and practical efficiency. However, the simplicity of GRPO can be deceptive—the vanilla GRPO algorithm has subtle issues that can hinder the RL training process, especially at scale. Solving the shortcomings of GRPO has become a popular research…

saved by

• Akira Yoshiyama
• Ishan Mukherjee

related reading

• State of RL for reasoning LLMs | A. Weersaweers.de
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• Why GRPO is Important and How it Worksghost.oxen.ai
• RLHF | John Lambertjohnwlambert.github.io
• Progressive Point Matchingprestonfu.com
• Interactive Visualization of RL Algorithms for LLM Trainingzcy233035.github.io
• DeepSeek-R1arxiv.org
• The Math Behind DeepSeek: A Deep Dive into Group Relative Policy Optimization (GRPO)medium.com
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• Lightweight Guide to understanding GRPO and RL principles - Musings of Muraligitlostmurali.com