Bite: How Deepseek R1 was trained

5 Minute Read on how Deepseek R1 was trained using Group Relative Policy Optimization (GRPO) and RL-focused multi-stage training approach.

Bite: How Deepseek R1 was trained January 17, 2025 4 minute read DeepSeek AI released DeepSeek-R1, an open model that rivals OpenAI's o1 in complex reasoning tasks, introduced using Group Relative Policy Optimization (GRPO) and RL-focused multi-stage training approach. Understanding Group Relative Policy Optimization (GRPO) Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm to improve the reasoning capabilities of LLMs. It was introduced in the DeepSeekMath paper in the context of mathematical reasoning. GRPO modifies the traditional Proximal Policy Optimization (P

related reading

• DeepSeek-R1arxiv.org
• Why GRPO is Important and How it Worksghost.oxen.ai
• The Math Behind DeepSeek: A Deep Dive into Group Relative Policy Optimization (GRPO)medium.com
• Open-R1: a fully open reproduction of DeepSeek-R1huggingface.co
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• DeepSeek R1's recipe to replicate o1 and the future of reasoning LMsinterconnects.ai
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• GitHub - deepseek-ai/DeepSeek-R1 · GitHubgithub.com
• The Illustrated DeepSeek-R1newsletter.languagemodels.co
• Interactive Visualization of RL Algorithms for LLM Trainingzcy233035.github.io
• GRPO Trainer · Hugging Facehuggingface.co
• State of RL for reasoning LLMs | A. Weersaweers.de