flâneur

[2507.13181] Spectral Bellman Method: Unifying Representation and Exploration in RL

arxiv.org · 8,022 words · saved by 1 readers

Abstract:Representation learning is critical to the empirical and theoretical success of reinforcement learning. However, many existing methods are induced from model-learning aspects, misaligning them with the RL task in hand. This work introduces the Spectral Bellman Method, a novel framework derived from the Inherent Bellman Error (IBE) condition. It aligns representation learning with the fundamental structure of Bellman updates across a \textit{space} of possible value functions, making it directly suited for value-based RL. Our key insight is a fundamental spectral relationship: under the zero-IBE condition, the transformation of a \textit{distribution} of value functions by the Bellman operator is intrinsically linked to the feature covariance structure. This connection yields a new, theoretically-grounded objective for learning state-action features that capture this Bellman-aligned covariance, requiring only a simple modification to existing algorithms. We demonstrate that our learned representations enable structured exploration by aligning feature covariance with Bellman dynamics, improving performance in hard-exploration and long-horizon tasks. Our framework naturally extends to multi-step Bellman operators, offering a principled path toward learning more powerful and structurally sound representations for value-based RL.

Published as a conference paper at ICLR 2026 S PECTRAL B ELLMAN M ETHOD : U NIFYING R EPRESENTATION AND E XPLORATION IN RL Ofir Nabati1,4∗, Bo Dai2 , Shie Mannor1,3 , Guy Tennenholtz4 1 Technion, 2 Google DeepMind, 3 Nvidia Research, 4 Google Research A BSTRACT…

saved by

related reading