[2605.06639] Recursive Agent Optimization

Abstract:We introduce Recursive Agent Optimization (RAO), a reinforcement learning approach for training recursive agents: agents that can spawn and delegate sub-tasks to new instantiations of themselves recursively. Recursive agents implement an inference-time scaling algorithm that naturally allows agents to scale to longer contexts and generalize to more difficult problems via divide-and-conquer. RAO provides a method to train models to best take advantage of such recursive inference, teaching agents when and how to delegate and communicate. We find that recursive agents trained in this way enjoy better training efficiency, can scale to tasks that go beyond the model's context window, generalize to tasks much harder than the ones the agent was trained on, and can enjoy reduced wall-clock time compared to single-agent systems.

View PDF Abstract:We introduce Recursive Agent Optimization (RAO), a reinforcement learning approach for training recursive agents: agents that can spawn and delegate sub-tasks to new instantiations of themselves recursively. Recursive agents implement an inference-time scaling algorithm that naturally allows agents to scale to longer contexts and generalize to more difficult problems via divide-and-conquer. RAO provides a method to train models to best take advantage of such recursive inference, teaching agents when and how to delegate and communicate. We find that recursive agents trained…

saved by

• Su P

related reading

• DeepSeek-R1arxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
• [2603.21972] Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipearxiv.org
• [2606.05555] Representation Learning Enables Scalable Multitask Deep Reinforcement Learningarxiv.org
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• [2602.05999] On the Role of Computation in Reinforcement Learningarxiv.org
• Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org