✳flâneur — a map of the web's best reading
IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMs
compute-optimal-rl-llm-scaling.github.io · 16,229 words · saved by 3 readers
IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMs (image URLs replaced with placeholders).
--> IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMs Figure 1: We study the compute-optimal RL for LLM along three axis: #parallel rollouts( n n n ), #problems per batch( B problem B_\text{problem} B problem ), and #sequential iterations( M M M ), where total rollout compute C = n × B problem × M C = n \times B_\text{problem} \times M C = n × B problem × M . We find that (1) optimal parallel rollouts per problem ( n n n ) grows with compute budget ( C C C ). (2) Easy and hard problems: similar scaling trends, but different mechanisms. (3) under fixed hardware
Explore this link on the map →saved by
related reading
- How to Explore to Scale RL Training of LLMs on Hard Problems? – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- Composer2.pdfcursor.com
- State of RL for reasoning LLMs | A. Weersaweers.de
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Q-learning is not yet scalableseohong.me
- How To Scale Your Modeljax-ml.github.io
- Scaling Laws for Value-Based RLvalue-scaling.github.io
- Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- How Well Does RL Scale? - Toby Ordtobyord.com