Q-learning is not yet scalable
Over the past few years, we've seen that next-token prediction scales, denoising diffusion scales, contrastive learning scales, and so on, all the way to the point where we can train models with billions of parameters with a scalable objective that can eat up as much data as we can throw at it. Then, what about reinforcement learning (RL)? Does RL also scale like all the other objectives? Apparently, it does. In 2016, RL achieved superhuman-level performance in games like Go and Chess. Now, RL is solving complex reasoning tasks in math and coding with large language models (LLMs). This is great. However, there is one important caveat: most of the current real-world successes of RL have been achieved with on-policy RL algorithms (e.g., REINFORCE, PPO, GRPO, etc.), which always require fresh, newly sampled rollouts from the current policy, and cannot reuse previous data (note: while PPO-like methods can technically reuse data to some (limited) degree, I'll classify them as on-policy RL,
Q-learning is not yet scalable Q-learning is not yet scalable Seohong Park UC Berkeley June 2025 Does RL scale? Over the past few years, we've seen that next-token prediction scales, denoising diffusion scales, contrastive learning scales, and so on, all the way to the point where we can train models with billions of parameters with a scalable objective that can eat up as much data as we can throw at it. Then, what about reinforcement learning (RL)? Does RL also scale like all the other objectives? Apparently, it does. In 2016, RL achieved superhuman-level performance in games like Go and Ches
saved by
related reading
- A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- How to Explore to Scale RL Training of LLMs on Hard Problems? – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
- Will scaling work?dwarkeshpatel.com
- Scaling is subtler than it seemsberen.io
- The upcoming GPT-3 moment for RL | Mechanize, Inc.mechanize.work
- State of RL for reasoning LLMs | A. Weersaweers.de
- RL without TD learningseohong.me
- You can’t imitation-learn how to continual-learn — LessWronglesswrong.com