IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMs
compute-optimal-rl-llm-scaling.github.io · 16,229 words · saved by 3 readers
IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMs (image URLs replaced with placeholders).
--> IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMs Figure 1: We study the compute-optimal RL for LLM along three axis: #parallel rollouts( n n n ), #problems per batch( B problem B_\text{problem} B problem ), and #sequential iterations( M M M ), where total rollout compute C = n × B problem × M C = n \times B_\text{problem} \times M C = n × B problem × M . We find that (1) optimal parallel rollouts per problem ( n n n ) grows with compute budget ( C C C ). (2) Easy and hard problems: similar scaling trends, but different mechanisms. (3) under fixed hardware
saved by
related reading
- How to Explore to Scale RL Training of LLMs on Hard Problems? – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
- Scaling Laws, Carefully | Lil'Loglilianweng.github.io
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Q-learning is not yet scalableseohong.me
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- How To Scale Your Modeljax-ml.github.io
- Lil'Loglilianweng.github.io
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- Scaling Laws for Value-Based RLvalue-scaling.github.io
- [2509.14786] Pre-training under infinite computearxiv.org
- Speeding up RL with high-leverage samples | Applied Computeappliedcompute.com
- State of RL for reasoning LLMs | A. Weersaweers.de