[2602.17616] Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
Abstract:Asynchronous reinforcement learning has become increasingly central to scaling LLM post-training, delivering major throughput gains by decoupling rollout generation from policy updates. However, widely used policy-gradient objectives such as REINFORCE and GRPO suffer under high asynchrony: stale rollouts produce heavy-tailed importance weights, so a small number of trajectories dominate updates and the policy-gradient estimator becomes markedly higher variance. Through systematic analysis on math, reasoning, and tool-use benchmarks, we find that this increasing variance is reliably predicted by collapsing effective sample size (ESS), which prior stabilization methods largely fail to address. Motivated by this diagnosis, we introduce $\textbf{V}$ariance $\textbf{C}$ontrolled $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{VCPO}$), a method that (i) dynamically scales the learning rate with ESS to dampen unreliable updates and (ii) applies a closed-form minimum-variance baseline for off-policy settings, without a critic model and adding minimal overhead. Empirically, across math and general reasoning benchmarks, this enables robustly stable asynchronous training compared to previous stabilization and algorithmic methods, even in highly off-policy regimes (128 steps off-policy). In a long-horizon, tool-use task, VCPO matches synchronous performance while delivering a 2.5$\times$ speedup in training time. Code is available at: this https URL
Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs Luke J. Huang 1 Zhuoyang Zhang 1 Qinghao Hu 1 Shang Yang 1 Song Han 1 2 Abstract as REINFORCE (Williams, 1992) and its recent variants such as GRPO (Shao et al., 2024). Asynchronous reinforcement learning has become…
saved by
related reading
- [2510.11696] QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMsarxiv.org
- [2504.13837] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?arxiv.org
- Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io
- [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- State of RL for reasoning LLMs | A. Weersaweers.de
- [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
- [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
- [2604.13010] Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationarxiv.org
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- [2604.00626] A Survey of On-Policy Distillation for Large Language Modelsarxiv.org
- Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co