How to scale RL to 10^26 FLOPs - by Jack Morris
TLDR: Reinforcement learning (RL) is the next training technique for building frontier-level AI models. To make it better, we need to train on more data. The current approach of scaling many environments simultaneously is messy and complicated. Instead, I propose we find a way to do next-token prediction on the Web using RL. This way, we learn to reason from general web data, instead of just math and code. I’ve spent a good part of the past year in denial. Thanks for reading! Subscribe for free to receive new posts and support my work. I was in denial because when OpenAI released o1, and explained their paradigm of test-time compute, I thought it was a good idea but mostly a way to get better performance out of models of fixed size. After all, letting models ‘think for longer’ by generating more tokens lets them do more internal computation. So I wasn’t that surprised that these new models, termed reasoning models, gave better answers. And I especially wasn’t surprised when I found out
How to scale RL to 10^26 FLOPs A roadmap for RL-ing LLMs on the entire Internet Jack Morris Jul 10, 2025 62 11 7 Share TLDR: Reinforcement learning (RL) is the next training technique for building frontier-level AI models. To make it better, we need to train on more data. The current approach of scaling many environments simultaneously is messy and complicated. Instead, I propose we find a way to do next-token prediction on the Web using RL. This way, we learn to reason from general web data, instead of just math and code. I’ve spent a good part of the past year in denial. Thanks for reading!
Explore this link on the map →related reading
- The Only Important Technology Is The Internet - Kevin Lukevinlu.ai
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
- State of RL for reasoning LLMs | A. Weersaweers.de
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
- DeepSeek-R1arxiv.org
- How Well Does RL Scale? - Toby Ordtobyord.com
- Scaling Reinforcement Learning: Environments, Reward Hacking, Agents, Scaling Datasemianalysis.com
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
- RL is even more information inefficient than you thoughtdwarkesh.com
- Just Ask for Generalization | Eric Jangevjang.com