[2603.21972] Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe

Abstract:Reinforcement Learning (RL) is essential for evolving Large Language Models (LLMs) into autonomous agents capable of long-horizon planning, yet a practical recipe for scaling RL in complex, multi-turn environments remains elusive. This paper presents a systematic empirical study using TravelPlanner, a challenging testbed requiring tool orchestration to satisfy multifaceted constraints. We decompose the agentic RL design space along 5 axes: reward shaping, model scaling, data composition, algorithm selection, and environmental stability. Our controlled experiments yield 7 key takeaways, e.g., (1) reward and algorithm choices are scale-dependent as smaller models benefit from staged rewards and enhanced exploration, whereas larger models converge efficiently with simpler dense rewards, (2) ~ 1K training samples with a balanced difficulty mixture mark a sweet spot for both in-domain and out-of-domain performance, and (3) environmental stability is critical to prevent policy degradation. Based on our distilled recipe, our RL-trained models achieve state-of-the-art performance on TravelPlanner, significantly outperforming leading LLMs.

# link_ootw0cwkrs.pdf ## Metadata - PDFFormatVersion=1.7 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=Xixi Wu; Qianguo Sun; Ruiyang Zhang; Chao Song; Junlong Wu; Yiyan Qi; Hong Cheng - Creator=arXiv GenPDF (tex2pdf:a6404ea) - Custom.DOI=https://doi.org/10.48550/arXiv.2603.21972 - Custom.License=http://arxiv.org/licenses/nonexclusive-distrib/1.0/ - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.28 (TeX Live 2025) kpathsea version 6.4.1 - Custom.arXivID=https://arxiv.org/abs/2603.21972v1

saved by

• Jeff Brown

related reading

• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• rlhfbook.com/book.pdfrlhfbook.com
• Composer2.pdfcursor.com
• The Era of Experience Paper.pdfstorage.googleapis.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• DeepSeek-R1arxiv.org
• A Taxonomy of RL Environments for LLM Agentsleehanchung.github.io
• Q-learning is not yet scalableseohong.me
• Explore | alphaXivalphaxiv.org
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Forge: Scalable Agent RL Framework and Algorithm - MiniMax News | MiniMaxminimax.io
• Rethinking RL Infra for Agents | B'Logbillxbf.github.io