Is Frontier Asynchronous RL Solved? — Luke J. Huang
A blog post by Luke J. Huang on whether frontier asynchronous RL is solved, covering policy lag, stability methods, open questions, and appendix notes.
Table of Contents Async RL has become the default for large-scale RL post-training. Frontier open-weights labs — GLM-5 , Ring 1T , DeepSeek V3.2 , Minimax M2.5 , Qwen 3.5 , Intellect-3 , Nemotron-3 Super , and Laguna-M.1 — report 2–3× faster throughput over synchronous pipelines, each with its own approach to keeping training stable. This is an attempt to survey that landscape: what does each lab do? What are the shared failure modes? Where do things currently stand? TL;DR async RL decouples rollout and training, giving 2-3x throughput, but the stale trajectories create off-policy instability
saved by
related reading
- Async RL in Pure JAXdivyamakkar0.github.io
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co
- State of RL for reasoning LLMs | A. Weersaweers.de
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- [2602.17616] Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMsarxiv.org
- RL at 1T Scale: prime-rl Performance Deep Diveprimeintellect.ai
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
- Journey to 2-second Inter-node RL Weight Transferle.qun.ch
- [2603.21383] PivotRL: High Accuracy Agentic Post-Training at Low Compute Costarxiv.org
- On-Policy Distillation: Promise, Pitfalls, and Prospectslouieworth.github.io