Thinking through how pretraining vs RL learn

Recently, people have been talking about how it takes way more FLOPs to get a single sample in RL than it does in supervised learning. In pretraining, you get a signal on every single token you train on. In RL, you have to unroll a whole thinking trajectory that’s 10s of 1000s of tokens long in order to get a single reward signal at the end (for example, did the unit test for my code pass/did I get the right answer to this math problem/etc). But this is only half the problem. Here’s a simple way to compare the learning efficiency of reinforcement learning versus supervised learning: Bits/FLOP = Samples/Flop * Bits/Sample. What I haven’t heard people talk about is the other term in our equation: Bits/Sample. And for most of training, the information density per sample is way way lower for RL. In supervised learning (aka pretraining), you’re just soaking up bits. Every token is a hint at the structure of language, and the mind crafting that language, and the world that mind is seeing. Ea

Blog RL is even more information inefficient than you thought And implications for RLVR progress Dwarkesh Patel Nov 17, 2025 268 28 16 Share Recently, people have been talking about how it takes way more FLOPs to get a single sample in RL than it does in supervised learning. In pretraining, you get a signal on every single token you train on. In RL, you have to unroll a whole thinking trajectory that’s 10s of 1000s of tokens long in order to get a single reward signal at the end (for example, did the unit test for my code pass/did I get the right answer to this math problem/etc). But this is o

saved by

• Lydia Nottingham
• Jason Hausenloy

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
• How to scale RL to 10^26 FLOPs - by Jack Morrisblog.jxmo.io
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• Just Ask for Generalization | Eric Jangevjang.com
• Reinforcement Learning for Knowledge Awareness – kalomaze's kalomazing blogkalomaze.bearblog.dev