[2602.19362] LLMs Can Learn to Reason Via Off-Policy RL

Abstract:Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.

LLMs Can Learn to Reason Via Off-Policy RL Daniel Ritter1 , Owen Oertell1,2 , Bradley Guo1 , Jonathan D. Chang2 , Kianté Brantley3 , Wen Sun2 1 Cornell University, 2 Databricks, 3 Harvard University Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algo- rithms, such as PPO or GRPO. However, policy lag from distributed…

saved by

• Katherine Driscoll
• Akira Yoshiyama

related reading

• PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• RLHF | John Lambertjohnwlambert.github.io
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• From REINFORCE to Dr. GRPOlancelqf.github.io
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
• DeepSeek-R1arxiv.org
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co