How to scale RL to 10^26 FLOPs - by Jack Morris

TLDR: Reinforcement learning (RL) is the next training technique for building frontier-level AI models. To make it better, we need to train on more data. The current approach of scaling many environments simultaneously is messy and complicated. Instead, I propose we find a way to do next-token prediction on the Web using RL. This way, we learn to reason from general web data, instead of just math and code. I’ve spent a good part of the past year in denial. Thanks for reading! Subscribe for free to receive new posts and support my work. I was in denial because when OpenAI released o1, and explained their paradigm of test-time compute, I thought it was a good idea but mostly a way to get better performance out of models of fixed size. After all, letting models ‘think for longer’ by generating more tokens lets them do more internal computation. So I wasn’t that surprised that these new models, termed reasoning models, gave better answers. And I especially wasn’t surprised when I found out

How to scale RL to 10^26 FLOPs A roadmap for RL-ing LLMs on the entire Internet Jack Morris Jul 10, 2025 62 11 7 Share TLDR: Reinforcement learning (RL) is the next training technique for building frontier-level AI models. To make it better, we need to train on more data. The current approach of scaling many environments simultaneously is messy and complicated. Instead, I propose we find a way to do next-token prediction on the Web using RL. This way, we learn to reason from general web data, instead of just math and code. I’ve spent a good part of the past year in denial. Thanks for reading!

related reading

• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• The upcoming GPT-3 moment for RL | Mechanize, Inc.mechanize.work
• Q-learning is not yet scalableseohong.me
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• RL is even more information inefficient than you thoughtdwarkesh.com
• RL Environments and RL for Science: Data Foundries and Multi-Agent Architecturesnewsletter.semianalysis.com
• How Well Does RL Scale? - Toby Ordtobyord.com
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com