GRPO++: Tricks for Making RL Actually Work

How to go from the vanilla GRPO algorithm to functional RL training at scale...

(from [1, 3, 4]) Recent research on large language models (LLMs) has been heavily focused on reasoning and reinforcement learning (RL). At the center of this research lies Group Relative Policy Optimization (GRPO) [13], the RL optimizer used to train most open-source reasoning models. The popularity of GRPO is enhanced by its conceptual simplicity and practical efficiency. However, the simplicity of GRPO can be deceptive—the vanilla GRPO algorithm has subtle issues that can hinder the RL training process, especially at scale. Solving the shortcomings of GRPO has become a popular research…

saved by

• Akira Yoshiyama
• Ishan Mukherjee

related reading

• State of RL for reasoning LLMs | A. Weersaweers.de
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• RLHF | John Lambertjohnwlambert.github.io
• Why GRPO is Important and How it Worksghost.oxen.ai
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• Interactive Visualization of RL Algorithms for LLM Trainingzcy233035.github.io
• Lightweight Guide to understanding GRPO and RL principles - Musings of Muraligitlostmurali.com
• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io