When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? | HTML5

Offline reinforcement learning (RL) algorithms can acquire effective policies by utilizing previously collected experience, without any online interaction. It is widely understood that offline RL is able to extract good policies even from highly suboptimal data, a scenario where imitation learning finds suboptimal solutions that do not improve over the demonstrator that generated the dataset. However, another common use case for practitioners is to learn from data that resembles demonstrations. In this case, one can choose to apply offline RL, but can also use behavioral cloning (BC) algorithms, which mimic a subset of the dataset via supervised learning. Therefore, it seems natural to ask: when can an offline RL method outperform BC with an equal amount of expert data, even when BC is a natural choice? To answer this question, we characterize the properties of environments that allow offline RL methods to perform better than BC methods, even when only provided with expert data. Additi

[2204.05618] When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning? ​​Aviral Kumar ∗,1,2 , Joey Hong ∗,1 , Anikait Singh 1 , Sergey Levine 1,2 1 Department of EECS, UC Berkeley 2 Google Research ( ∗ Equal Contribution) {aviralk, joey_hong}@berkeley.edu Abstract Offline reinforcement learning (RL) algorithms can acquire effective policies by utilizing previously collected experience, without any online interaction. It is widely understood that offline RL is able to extract good policies even fro

related reading

• Should I Use Offline RL or Imitation Learning? – The Berkeley Artificial Intelligence Research Blogbair.berkeley.edu
• 2109.10813.pdfarxiv.org
• [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org
• [2406.09329] Is Value Learning Really the Main Bottleneck in Offline RL?ar5iv.labs.arxiv.org
• [2005.01643] Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemsar5iv.labs.arxiv.org
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• Behavioral cloning mysteryseohong.me
• [2307.04354] Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Dataar5iv.labs.arxiv.org
• Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• [2201.11861] The Challenges of Exploration for Offline Reinforcement Learningar5iv.labs.arxiv.org
• [2506.22401] Exploration from a Primal-Dual Lens: Value-Incentivized Actor-Critic Methods for Sample-Efficient Online RLarxiv.org