Just make the straw bigger

When life gives you one bit per lemon, how many of them will make for a lemonade?

Special thanks go to Tommaso Furlanello, Neev Parikh and @hallerite for reading earlier versions of this post and pushing me to write this. In LoRA without regret , Schulman et al. informally claim that supervised learning can deliver O ( #tokens ) O(\text{\#tokens}) O ( #tokens ) bits of supervision per episode, whereas policy-gradient RL gets only O ( 1 ) O(1) O ( 1 ) bits per episode because the advantage signal is effectively a single scalar per episode. Karpathy, too, at Dwarkesh’s podcast, compared RL to “ sucking supervision through a straw.” The informal claim here is that low-bandwidt

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• State of RL for reasoning LLMs | A. Weersaweers.de
• RL is even more information inefficient than you thoughtdwarkesh.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectoriesarxiv.org
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Understanding R1-Zero-Like Training: A Critical Perspectivearxiv.org
• Speeding up RL with high-leverage samples | Applied Computeappliedcompute.com
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• From REINFORCE to Dr. GRPOlancelqf.github.io