Is Frontier Asynchronous RL Solved? — Luke J. Huang
A blog post by Luke J. Huang on whether frontier asynchronous RL is solved, covering policy lag, stability methods, open questions, and appendix notes.
Table of Contents Async RL has become the default for large-scale RL post-training. Frontier open-weights labs — GLM-5 , Ring 1T , DeepSeek V3.2 , Minimax M2.5 , Qwen 3.5 , Intellect-3 , Nemotron-3 Super , and Laguna-M.1 — report 2–3× faster throughput over synchronous pipelines, each with its own approach to keeping training stable. This is an attempt to survey that landscape: what does each lab do? What are the shared failure modes? Where do things currently stand? TL;DR async RL decouples rollout and training, giving 2-3x throughput, but the stale trajectories create off-policy instability
saved by
related reading
- Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co
- Async RL in Pure JAXdivyamakkar0.github.io
- State of RL for reasoning LLMs | A. Weersaweers.de
- [2602.17616] Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMsarxiv.org
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
- Progressive Point Matchingprestonfu.com
- RL at 1T Scale: prime-rl Performance Deep Diveprimeintellect.ai
- Infini-AI-Lab on X: "We’re excited to release 𝐀𝐬𝐭𝐫𝐚𝐅𝐥𝐨𝐰, an open-source, dataflow-oriented RL system for training multi-agentic and multi-policy LLMs. 🚀 Built for scalable, flexible, and efficient agent RL, AstraFlow natively enables: ⚡ 𝟐.𝟕× 𝐟𝐚𝐬𝐭𝐞𝐫 𝐦𝐮𝐥𝐭𝐢-𝐩𝐨𝐥𝐢𝐜𝐲 https://t.co/JVthM8iHur" / Xx.com
- RLHF Bookrlhfbook.com