[2603.21383] PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost

Abstract:Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL, a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots: informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while maximally preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves +4.17% higher in-domain accuracy on average across four agentic domains, and +10.04% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4x fewer rollout turns. PivotRL is adopted by NVIDIA's Nemotron-3-Super-120B-A12B, acting as the workhorse in production-scale agentic post-training.

2026-3-24 PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost Junkeun Yi*,1 Damon Mosk-Aoyama*,1 Baihe Huang*,2 Ritu Gala1 Charles Wang1 Sugam Dipak Devare1 Khushi Bhardwaj1 Abhibha Gupta1 Oleksii Kuchaiev1 Jiantao Jiao**,1,2 Jian Zhang**,1 Venkat Srinivasan**,†,1 1 2 NVIDIA…

saved by

• Akira Yoshiyama

related reading

• [2606.05555] Representation Learning Enables Scalable Multitask Deep Reinforcement Learningarxiv.org
• [2602.11399] Can We Really Learn One Representation to Optimize All Rewards?arxiv.org
• [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org
• [2602.05999] On the Role of Computation in Reinforcement Learningarxiv.org
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• [2603.21972] Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipearxiv.org
• [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org
• [2606.10346] Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learningarxiv.org
• [2506.19733] Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?arxiv.org