GRPO is terrible — LessWrong

I once tried teaching a simulated 3D humanoid how to dunk[1]using RL - truly the most effective use of my time. If I took a single thing away from that, it was that designing the best reward function is equivalent to believing your agent is conscious and has the single goal to destroy your project. My point is, RL is already terrible in practice. Then additionally throwing intermediate rewards out the window and overly relying on the most inefficient part of modern LLMs, their autoregressive inference[2], doesn't exactly seem like the play - somehow it is though. The first attempts did try exactly this - basically an additional model that takes in one 'reasoning step' and spits out a number - the reward. The problem is that we simply don't have any pretraining data for such a reward model. Generating your own data is expensive[3]and not comparable to any pretraining scale. There's also a whole different problem - expecting such model to be feasible in the first place[4]: even humans ve

x GRPO is terrible — LessWrong AI Frontpage 4 GRPO is terrible by RobinHa 1st Dec 2025 Linkpost for robinhaselhorst.com 6 min read 2 4 An on-policy, sample-efficient NLP post-training approach not requiring verification The current state I once tried teaching a simulated 3D humanoid how to dunk [1] using RL - truly the most effective use of my time. If I took a single thing away from that, it was that designing the best reward function is equivalent to believing your agent is conscious and has the single goal to destroy your project. My point is, RL is already terrible in practice. Then additi

related reading

• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• Pedagogical RL: Teaching Models to Teach Themselves from Privileged Information - Noah Ziemsnoahziems.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Why GRPO is Important and How it Worksghost.oxen.ai
• SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• DeepSeek-R1arxiv.org
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com