flâneur

[2602.17616] Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs

arxiv.org · 5,636 words · saved by 1 readers

Abstract:Asynchronous reinforcement learning has become increasingly central to scaling LLM post-training, delivering major throughput gains by decoupling rollout generation from policy updates. However, widely used policy-gradient objectives such as REINFORCE and GRPO suffer under high asynchrony: stale rollouts produce heavy-tailed importance weights, so a small number of trajectories dominate updates and the policy-gradient estimator becomes markedly higher variance. Through systematic analysis on math, reasoning, and tool-use benchmarks, we find that this increasing variance is reliably predicted by collapsing effective sample size (ESS), which prior stabilization methods largely fail to address. Motivated by this diagnosis, we introduce $\textbf{V}$ariance $\textbf{C}$ontrolled $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{VCPO}$), a method that (i) dynamically scales the learning rate with ESS to dampen unreliable updates and (ii) applies a closed-form minimum-variance baseline for off-policy settings, without a critic model and adding minimal overhead. Empirically, across math and general reasoning benchmarks, this enables robustly stable asynchronous training compared to previous stabilization and algorithmic methods, even in highly off-policy regimes (128 steps off-policy). In a long-horizon, tool-use task, VCPO matches synchronous performance while delivering a 2.5$\times$ speedup in training time. Code is available at: this https URL

Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs Luke J. Huang 1 Zhuoyang Zhang 1 Qinghao Hu 1 Shang Yang 1 Song Han 1 2 Abstract as REINFORCE (Williams, 1992) and its recent variants such as GRPO (Shao et al., 2024). Asynchronous reinforcement learning has become…

saved by

related reading