Speeding up RL with high-leverage samples | Applied Compute

We learn the most from a problem when we struggle but eventually solve it. We learn what works and what doesn’t, preparing us for the next one. But not all parts of this process teach us the same amount: the rare flashes of insight teach the most. Unlike humans, who attempt a problem once and then solve it or give up, language models generate many independent attempts, or rollouts. Naive reinforcement learning algorithms assume that all of these attempts contain the same amount of information, and thus use all of them to update the model. But just as humans learn the most from rare insight, models learn the most from rare rollouts. If we give a model one hundred attempts at a difficult math problem and it solves it only ten times, those ten attempts teach the model far more than the ninety others. In this blog, we formalize this intuition and find that in a problem with a 10% success rate like the one above, each successful rollout is 81 times more valuable than a failed one! More gene

We learn the most from a problem when we struggle but eventually solve it. We learn what works and what doesn’t, preparing us for the next one. But not all parts of this process teach us the same amount: the rare flashes of insight teach the most. Unlike humans, who attempt a problem once and then solve it or give up, language models generate many independent attempts, or rollouts. Naive reinforcement learning algorithms assume that all of these attempts contain the same amount of information, and thus use all of them to update the model. But just as humans learn the most from rare insight, mo

saved by

• Rajan Agarwal

related reading

• State of RL for reasoning LLMs | A. Weersaweers.de
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
• From REINFORCE to Dr. GRPOlancelqf.github.io
• RLHF Bookrlhfbook.com
• Debugging Reinforcement Learning Systemsandyljones.com
• Lightweight Guide to understanding GRPO and RL principles - Musings of Muraligitlostmurali.com
• RL is even more information inefficient than you thoughtdwarkesh.com
• Why GRPO is Important and How it Worksghost.oxen.ai
• Understanding R1-Zero-Like Training: A Critical Perspectivearxiv.org