GRPO Trainer

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Overview TRL supports the GRPO Trainer for training language models, as described in the paper DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models by Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo. The abstract from the paper is the following: Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from…

related reading

• Why GRPO is Important and How it Worksghost.oxen.ai
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• The Math Behind DeepSeek: A Deep Dive into Group Relative Policy Optimization (GRPO)medium.com
• DeepSeek-R1arxiv.org
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelsarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• Interactive Visualization of RL Algorithms for LLM Trainingzcy233035.github.io
• GitHub - FusionBrainLab/Vision_GRPOgithub.com
• Tutorial: Train your own Reasoning model with GRPO | Unsloth Documentationdocs.unsloth.ai
• Lightweight Guide to understanding GRPO and RL principles - Musings of Muraligitlostmurali.com
• [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org