When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? | HTML5

Offline reinforcement learning (RL) algorithms can acquire effective policies by utilizing previously collected experience, without any online interaction. It is widely understood that offline RL is able to extract good policies even from highly suboptimal data, a scenario where imitation learning finds suboptimal solutions that do not improve over the demonstrator that generated the dataset. However, another common use case for practitioners is to learn from data that resembles demonstrations. In this case, one can choose to apply offline RL, but can also use behavioral cloning (BC) algorithms, which mimic a subset of the dataset via supervised learning. Therefore, it seems natural to ask: when can an offline RL method outperform BC with an equal amount of expert data, even when BC is a natural choice? To answer this question, we characterize the properties of environments that allow offline RL methods to perform better than BC methods, even when only provided with expert data. Additi

[2204.05618] When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? ​​Aviral Kumar ∗,1,2 , Joey Hong ∗,1 , Anikait Singh 1 , Sergey Levine 1,2 1 Department of EECS, UC Berkeley 2 Google Research ( ∗ Equal Contribution) {aviralk, joey_hong}@berkeley.edu Abstract Offline reinforcement learning (RL) algorithms can acquire effective policies by utilizing previously collected experience, without any online interaction. It is widely understood that offline RL is able to extract good policies even fro

related reading

• Should I Use Offline RL or Imitation Learning? – The Berkeley Artificial Intelligence Research Blogbair.berkeley.edu
• [2406.09329] Is Value Learning Really the Main Bottleneck in Offline RL?ar5iv.labs.arxiv.org
• [2005.01643] Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemsar5iv.labs.arxiv.org
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• [2201.11861] The Challenges of Exploration for Offline Reinforcement Learningar5iv.labs.arxiv.org
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• [2006.03647] Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationar5iv.labs.arxiv.org
• Reinforcement Learning via Implicit Imitation Guidancearxiv.org
• [2307.04354] Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Dataar5iv.labs.arxiv.org
• State of Robot Learning, December 2025vedder.io
• Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuationarxiv.org
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com