Reinforcement Learning with Action Chunking | HTML5

We present Q-chunking, a simple yet effective recipe for improving reinforcement learning (RL) algorithms for long-horizon, sparse-reward tasks. Our recipe is designed for the offline-to-online RL setting, where the goal is to leverage an offline prior dataset to maximize the sample-efficiency of online learning. Effective exploration and sample-efficient learning remain central challenges in this setting, as it is not obvious how the offline data should be utilized to acquire a good exploratory policy. Our key insight is that action chunking, a technique popularized in imitation learning where sequences of future actions are predicted rather than a single action at each timestep, can be applied to temporal difference (TD)-based RL methods to mitigate the exploration challenge. Q-chunking adopts action chunking by directly running RL in a ‘chunked’ action space, enabling the agent to (1) leverage temporally consistent behaviors from offline data for more effective online exploration an

Reinforcement Learning with Action Chunking Qiyang Li, Zhiyuan Zhou, Sergey Levine UC Berkeley {qcli,zhiyuan_zhou,svlevine}@eecs.berkeley.edu Abstract We present Q-chunking , a simple yet effective recipe for improving reinforcement learning (RL) algorithms for long-horizon, sparse-reward tasks. Our recipe is designed for the offline-to-online RL setting, where the goal is to leverage an offline prior dataset to maximize the sample-efficiency of online learning. Effective exploration and sample-efficient learning remain central challenges in this setting, as it is not obvious how the offline d

related reading

• Reinforcement Learning with Action Chunkingarxiv.org
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org
• Q-learning is not yet scalableseohong.me
• [2507.09061] Action Chunking and Exploratory Data Collection Yield Exponential Improvements in Behavior Cloning for Continuous Controlarxiv.org
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• 2109.10813.pdfarxiv.org
• c21f4ce780c5c9d774f79841b81fdc6d-Paper.pdfproceedings.neurips.cc
• [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• [2307.04354] Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Dataar5iv.labs.arxiv.org
• [2606.05555] Representation Learning Enables Scalable Multitask Deep Reinforcement Learningarxiv.org