Speeding up RL with high-leverage samples | Applied Compute

We learn the most from a problem when we struggle but eventually solve it. We learn what works and what doesn’t, preparing us for the next one. But not all parts of this process teach us the same amount: the rare flashes of insight teach the most. Unlike humans, who attempt a problem once and then solve it or give up, language models generate many independent attempts, or rollouts. Naive reinforcement learning algorithms assume that all of these attempts contain the same amount of information, and thus use all of them to update the model. But just as humans learn the most from rare insight, models learn the most from rare rollouts. If we give a model one hundred attempts at a difficult math problem and it solves it only ten times, those ten attempts teach the model far more than the ninety others. In this blog, we formalize this intuition and find that in a problem with a 10% success rate like the one above, each successful rollout is 81 times more valuable than a failed one! More gene

We learn the most from a problem when we struggle but eventually solve it. We learn what works and what doesn’t, preparing us for the next one. But not all parts of this process teach us the same amount: the rare flashes of insight teach the most. Unlike humans, who attempt a problem once and then solve it or give up, language models generate many independent attempts, or rollouts. Naive reinforcement learning algorithms assume that all of these attempts contain the same amount of information, and thus use all of them to update the model. But just as humans learn the most from rare insight, mo

saved by

• Rajan Agarwal
• Julian Quevedo

related reading

• State of RL for reasoning LLMs | A. Weersaweers.de
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• Interactive Visualization of RL Algorithms for LLM Trainingzcy233035.github.io
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• RL Post-Training on Macs | Pluralis Researchpluralis.ai
• Just make the straw bigger | Joan Veljajoanvelja.com
• From REINFORCE to Dr. GRPOlancelqf.github.io
• RLHF Bookrlhfbook.com