Thinking through how pretraining vs RL learn

Recently, people have been talking about how it takes way more FLOPs to get a single sample in RL than it does in supervised learning. In pretraining, you get a signal on every single token you train on. In RL, you have to unroll a whole thinking trajectory that’s 10s of 1000s of tokens long in order to get a single reward signal at the end (for example, did the unit test for my code pass/did I get the right answer to this math problem/etc). But this is only half the problem. Here’s a simple way to compare the learning efficiency of reinforcement learning versus supervised learning: Bits/FLOP = Samples/Flop * Bits/Sample. What I haven’t heard people talk about is the other term in our equation: Bits/Sample. And for most of training, the information density per sample is way way lower for RL. In supervised learning (aka pretraining), you’re just soaking up bits. Every token is a hint at the structure of language, and the mind crafting that language, and the world that mind is seeing. Ea

Recently, people have been talking about how it takes way more FLOPs to get a single sample in RL than it does in supervised learning. In pretraining, you get a signal on every single token you train on. In RL, you have to unroll a whole thinking trajectory that’s 10s of 1000s of tokens long in order to get a single reward signal at the end (for example, did the unit test for my code pass/did I get the right answer to this math problem/etc). But this is only half the problem. Here’s a simple way to compare the learning efficiency of reinforcement learning versus supervised learning:…

related reading

• RL is even more information inefficient than you thoughtdwarkesh.com
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• Contra Dwarkesh on RL sample-efficiency via information theorynewsletter.danielpaleka.com
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Speeding up RL with high-leverage samples | Applied Computeappliedcompute.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• ML Mentorship: Some Q/A about RLevjang.com
• How to scale RL to 10^26 FLOPs - by Jack Morrisblog.jxmo.io
• Q-learning is not yet scalableseohong.me