Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

TL;DR -- For those of you who don't have time to read 5,000 words about async RL plumbing (we get it, you have models to train): The problem: In synchronous RL (reinforcement learning) training, data generation (model inference to create data samples) dominates wall-clock time -- a single batch of 32K-token rollouts on a 32B (32-billion parameter) model can take hours, while the GPUs used for training remain idle. The solution everyone converged on: Disaggregate (separate) inference and training onto different GPU pools, connect them with a rollout buffer (temporary storage for model…

saved by

• Dhruv Sheth
• Akira Yoshiyama
• Ishan Mukherjee

related reading

• State of RL for reasoning LLMs | A. Weersaweers.de
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io
• RL Post-Training on Macs | Pluralis Researchpluralis.ai
• Async RL in Pure JAXdivyamakkar0.github.io
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• The upcoming GPT-3 moment for RL | Mechanize, Inc.mechanize.work
• RL at 1T Scale: prime-rl Performance Deep Diveprimeintellect.ai
• Composer2.pdfcursor.com
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com