[2605.03327] DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment
Abstract:Reinforcement learning is crucial for aligning large language models to perform complex reasoning tasks. However, current algorithms such as Group Relative Policy Optimization suffer from coarse grained, sequence level credit assignment, which severely struggles to isolate pivotal reasoning steps within long Chain of Thought generations. Furthermore, the standard unbounded Kullback Leibler divergence penalty induces severe gradient instability and mode seeking conservatism, ultimately stifling the discovery of novel reasoning trajectories. To overcome these limitations, we introduce Distribution Guided Policy Optimization, a novel critic free reinforcement learning framework that reinterprets distribution deviation as a guiding signal rather than a rigid penalty.
# link_ulpttedz7s.pdf ## Metadata - PDFFormatVersion=1.7 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=Hongbo Jin; Rongpeng Zhu; Zhongjing Du; Xu Jiang; Jingqi Tian; Qiaoman Zhang; Jiayu Ding - Creator=arXiv GenPDF (tex2pdf:a6404ea) - Custom.DOI=https://doi.org/10.48550/arXiv.2605.03327 - Custom.License=http://arxiv.org/licenses/nonexclusive-distrib/1.0/ - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.28 (TeX Live 2025) kpathsea version 6.4.1 - Custom.arXivID=https://arxiv.org/abs/2605.
saved by
related reading
- On the Geometry of On-Policy Distillationarxiv.org
- [2604.00626] A Survey of On-Policy Distillation for Large Language Modelsarxiv.org
- State of RL for reasoning LLMs | A. Weersaweers.de
- GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
- A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
- DeepSeek-R1arxiv.org
- Why GRPO is Important and How it Worksghost.oxen.ai
- A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
- The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
- Interactive Visualization of RL Algorithms for LLM Trainingzcy233035.github.io