[2606.05555] Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
Abstract:Scaling reinforcement learning (RL) to diverse multitask settings remains a central challenge. While recent advances in model-based RL achieve strong performance, they rely on planning and complex training pipelines, making it unclear which components are essential for scalability. We revisit this question and argue that the primary driver of scalable multitask RL is not model-based control, but \emph{representation learning}. In particular, we show that combining predictive, model-based representations with high-capacity value function approximation is sufficient to achieve strong performance, even without planning. We evaluate a simple model-free algorithm, MR.Q, coupled with auxiliary predictive objectives into a scalable actor-critic architecture. This approach outperforms a recent world-model-based method and a range of deep RL baselines across a diverse suite of multitask continuous control tasks, while significantly reducing computational overhead and improving wall-clock efficiency. We observe consistent improvements with increased model capacity and show through ablations that predictive representation learning is critical for performance.
Representation Learning Enables Scalable Multitask Deep Reinforcement Learning Johan Obando-Ceron1,2 Lu Li1,2 Scott Fujimoto3 Pierre-Luc Bacon1,2 Aaron Courville1,2,4 Pablo Samuel Castro1,2,5 1 Mila – Québec AI Institute 2 Université de Montréal 3 McGill University…
saved by
related reading
- [2602.05999] On the Role of Computation in Reinforcement Learningarxiv.org
- [2602.11399] Can We Really Learn One Representation to Optimize All Rewards?arxiv.org
- [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org
- [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
- [2507.13181] Spectral Bellman Method: Unifying Representation and Exploration in RLarxiv.org
- [2603.21383] PivotRL: High Accuracy Agentic Post-Training at Low Compute Costarxiv.org
- [1710.10044] Distributional Reinforcement Learning with Quantile Regressionarxiv.org
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Q-learning is not yet scalableseohong.me
- 2006.10742arxiv.org
- [2205.13147] Matryoshka Representation Learningarxiv.org
- [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org