[2602.05999] On the Role of Computation in Reinforcement Learning
Abstract:How does the amount of compute available to a reinforcement learning (RL) policy affect its learning? Can policies using a fixed amount of parameters, still benefit from additional compute? The standard RL framework does not provide a language to answer these questions formally. Empirically, deep RL policies are often parameterized as neural networks with static architectures, conflating the amount of compute and the number of parameters. In this paper, we formalize compute bounded policies and prove that policies which use more compute can solve problems and generalize to longer-horizon tasks that are outside the scope of policies with less compute. Building on prior work in algorithmic learning and model-free planning, we propose a minimal architecture that can use a variable amount of compute. Our experiments complement our theory. On a set 31 different tasks spanning online and offline RL, we show that $(1)$ this architecture achieves stronger performance simply by using more compute, and $(2)$ stronger generalization on longer-horizon test tasks compared to standard feedforward networks or deep residual network using up to 5 times more parameters.
On the Role of Computation in Reinforcement Learning Raj Ghugare 1 Michał Bortkiewicz * 1 2 Alicja Ziarko * 1 3 4 5 Benjamin Eysenbach 1 Abstract action of where to move the right foot is easy, but figur- ing out how to efficiently pack furniture in a truck requires How does…
saved by
related reading
- [2606.05555] Representation Learning Enables Scalable Multitask Deep Reinforcement Learningarxiv.org
- [2602.11399] Can We Really Learn One Representation to Optimize All Rewards?arxiv.org
- [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
- [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org
- f514cec81cb148559cf475e7426eed5e-Paper.pdfproceedings.neurips.cc
- [2306.01157] Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingarxiv.org
- [2603.21383] PivotRL: High Accuracy Agentic Post-Training at Low Compute Costarxiv.org
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
- Quantifying Generalization in Reinforcement Learningarxiv.org