IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMs
compute-optimal-rl-llm-scaling.github.io · 16,229 words · saved by 3 readers
IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMs (image URLs replaced with placeholders).
--> IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMs Figure 1: We study the compute-optimal RL for LLM along three axis: #parallel rollouts( n n n ), #problems per batch( B problem B_\text{problem} B problem ), and #sequential iterations( M M M ), where total rollout compute C = n × B problem × M C = n \times B_\text{problem} \times M C = n × B problem × M . We find that (1) optimal parallel rollouts per problem ( n n n ) grows with compute budget ( C C C ). (2) Easy and hard problems: similar scaling trends, but different mechanisms. (3) under fixed hardware
saved by
related reading
- How to Explore to Scale RL Training of LLMs on Hard Problems? – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- Composer2.pdfcursor.com
- Scaling Laws, Carefully | Lil'Loglilianweng.github.io
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- State of RL for reasoning LLMs | A. Weersaweers.de
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- Scaling Laws for Value-Based RLvalue-scaling.github.io
- The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
- Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
- Value-Based Deep RL Scales Predictablyarxiv.org