Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

TL;DR -- For those of you who don't have time to read 5,000 words about async RL plumbing (we get it, you have models to train): The problem: In synchronous RL (reinforcement learning) training, data generation (model inference to create data samples) dominates wall-clock time -- a single batch of 32K-token rollouts on a 32B (32-billion parameter) model can take hours, while the GPUs used for training remain idle. The solution everyone converged on: Disaggregate (separate) inference and training onto different GPU pools, connect them with a rollout buffer (temporary storage for model…

saved by

• Dhruv Sheth
• Akira Yoshiyama
• Ishan Mukherjee

related reading

• State of RL for reasoning LLMs | A. Weersaweers.de
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io
• Debugging Reinforcement Learning Systemsandyljones.com
• RL Post-Training on Macs | Pluralis Researchpluralis.ai
• [2603.21383] PivotRL: High Accuracy Agentic Post-Training at Low Compute Costarxiv.org
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• A Call to Build Models Like We Build Open-Source Softwarecolinraffel.com