How can LLM RL Work Despite Information-Theoretic Inefficiency
Epistemic Status: Obviously speculative and maybe obvious. The success of RL in LLMs has been puzzling me for a while. People have developed various information-theoretic style arguments by which they argue that RL is extremely informationally inefficient compared to pretraining, that it can only impart a tiny amount of bits,...
Epistemic Status : Obviously speculative and maybe obvious. The success of RL in LLMs has been puzzling me for a while. People have developed various information-theoretic style arguments by which they argue that RL is extremely informationally inefficient compared to pretraining , that it can only impart a tiny amount of bits, and that it can only bring out behaviours that are already in the base models etc. The basic intuition here is extremely obvious and essentially falls immediately out of the formulation of the two methods. Pretraining (and SFT etc) compute a loss on every token so every
saved by
- Ratan Kaliani
- Uzay Girit
- Vincent Cheng
- Dhruv Sheth
- Andrew Cai
- Emil Ryd
- Brady Gho
- Ishaan Panigrahi
- Al-Ekram Elahee Hridoy
- Chris Shi
- Agape Keleta
- Nathan Chen
related reading
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- RL is even more information inefficient than you thoughtdwarkesh.com
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
- Scaling is subtler than it seemsberen.io
- State of RL for reasoning LLMs | A. Weersaweers.de
- RL is even more information inefficient than you thoughtsubstack.com
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Compression and Intelligencegreene.sh
- You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
- Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org