[2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RL

Abstract:Online reinforcement learning (RL) with complex function approximations such as transformers and deep neural networks plays a significant role in the modern practice of artificial intelligence. Despite its popularity and importance, balancing the fundamental trade-off between exploration and exploitation remains a long-standing challenge; in particular, we are still in lack of efficient and practical schemes that are backed by theoretical performance guarantees. Motivated by recent developments in exploration via optimistic regularization, this paper provides an interpretation of the principle of optimism through the lens of primal-dual optimization. From this fresh perspective, we set forth a new value-incentivized actor-critic (VAC) method, which optimizes a single easy-to-optimize objective integrating exploration and exploitation -- it promotes state-action and policy estimates that are both consistent with collected data transitions and result in higher value functions. Theoretically, the proposed VAC method has near-optimal regret guarantees under linear Markov decision processes (MDPs) in both finite-horizon and infinite-horizon settings, which can be extended to the general function approximation setting under appropriate assumptions.

Abstract:Online reinforcement learning (RL) with complex function approximations such as transformers and deep neural networks plays a significant role in the modern practice of artificial intelligence. Despite its popularity and importance, balancing the fundamental trade-off between exploration and exploitation remains a long-standing challenge; in particular, we are still in lack of efficient and practical schemes that are backed by theoretical performance guarantees. Motivated by recent developments in exploration via optimistic regularization, this paper provides an interpretation of the

saved by

• Akira Yoshiyama

related reading

• [2507.13181] Spectral Bellman Method: Unifying Representation and Exploration in RLarxiv.org
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• Key Papers in Deep RL - Spinning Up documentationspinningup.openai.com
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• Soft Actor-Critic - Spinning Up documentationspinningup.openai.com
• pistar06.pdfpi.website
• π*0.6: a VLA That Learns From Experiencephysicalintelligence.company