[2606.10346] Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learning

Abstract:Reinforcement learning has become a key paradigm for eliciting reasoning abilities in large language models, where exploration is crucial for discovering effective solution trajectories. Existing exploration methods typically encourage diversity in semantic or gradient spaces, without distinguishing what drives this diversity. A trajectory may appear novel because it follows a new reasoning process, or because it varies memorized patterns and shortcuts. Rewarding both cases equally may steer exploration toward memorization rather than genuine reasoning improvement. In this paper, we propose DiRL, a Direction-Aware Reinforcement Learning framework that anchors exploration to an internal reasoning-memorization direction of the policy. Specifically, DiRL extracts this direction from model representations, constructs direction-weighted gradient features to characterize rollout updates, and shapes rewards to amplify reasoning-aligned exploration while suppressing memorization-aligned variations. DiRL integrates seamlessly into standard Group Relative Policy Optimization (GRPO). Extensive experiments on mathematical and general reasoning benchmarks demonstrate the effectiveness of DiRL, showing significant improvements over various existing exploration methods.

# link_1no60duzhbi.pdf ## Metadata - PDFFormatVersion=1.7 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=Jiangnan Xia; Yucheng Shi; Yu Yang; Kishan Panaganti; Zhenwen Liang; Ninghao Liu - Creator=arXiv GenPDF (tex2pdf:a6404ea) - Custom.DOI=https://doi.org/10.48550/arXiv.2606.10346 - Custom.License=http://creativecommons.org/licenses/by/4.0/ - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.28 (TeX Live 2025) kpathsea version 6.4.1 - Custom.arXivID=https://arxiv.org/abs/2606.10346v1 - Produ

saved by

• Christina Lee

related reading

• [2605.15156] MeMo: Memory as a Modelarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• DeepSeek-R1arxiv.org
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• The State of Reinforcement Learning for LLM Reasoningsebastianraschka.com
• [2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningarxiv.org
• LLM Resourcesforrestbicker.com
• [2605.03327] DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignmentarxiv.org
• Explore | alphaXivalphaxiv.org
• [2506.01939] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningarxiv.org
• Unlocking the Working Memory of Large Language Models for Latent Reasoningarxiv.org
• You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectoriesarxiv.org