When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? | HTML5

Offline reinforcement learning (RL) algorithms can acquire effective policies by utilizing previously collected experience, without any online interaction. It is widely understood that offline RL is able to extract good policies even from highly suboptimal data, a scenario where imitation learning finds suboptimal solutions that do not improve over the demonstrator that generated the dataset. However, another common use case for practitioners is to learn from data that resembles demonstrations. In this case, one can choose to apply offline RL, but can also use behavioral cloning (BC) algorithms, which mimic a subset of the dataset via supervised learning. Therefore, it seems natural to ask: when can an offline RL method outperform BC with an equal amount of expert data, even when BC is a natural choice? To answer this question, we characterize the properties of environments that allow offline RL methods to perform better than BC methods, even when only provided with expert data. Additi

[2204.05618] When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? ​​Aviral Kumar ∗,1,2 , Joey Hong ∗,1 , Anikait Singh 1 , Sergey Levine 1,2 1 Department of EECS, UC Berkeley 2 Google Research ( ∗ Equal Contribution) {aviralk, joey_hong}@berkeley.edu Abstract Offline reinforcement learning (RL) algorithms can acquire effective policies by utilizing previously collected experience, without any online interaction. It is widely understood that offline RL is able to extract good policies even fro

related reading

• Should I Use Offline RL or Imitation Learning? – The Berkeley Artificial Intelligence Research Blogbair.berkeley.edu
• [2406.09329] Is Value Learning Really the Main Bottleneck in Offline RL?ar5iv.labs.arxiv.org
• [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org
• 2109.10813.pdfarxiv.org
• [2005.01643] Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemsar5iv.labs.arxiv.org
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• Behavioral cloning mysteryseohong.me
• State of Robot Learning, December 2025vedder.io
• Value estimation with finite datamcgill.scholaris.ca
• [2201.11861] The Challenges of Exploration for Offline Reinforcement Learningar5iv.labs.arxiv.org
• Reinforcement Learning via Implicit Imitation Guidancearxiv.org
• [2006.03647] Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationar5iv.labs.arxiv.org