Provably Good Batch Reinforcement Learning Without Great Exploration | HTML5

Batch reinforcement learning (RL) is important to apply RL algorithms to many high stakes tasks. Doing batch RL in a way that yields a reliable new policy in large domains is challenging: a new decision policy may visit states and actions outside the support of the batch data, and function approximation and optimization with limited samples can further increase the potential of learning policies with overly optimistic estimates of their future performance. Recent algorithms have shown promise but can still be overly optimistic in their expected outcomes. Theoretical work that provides strong guarantees on the performance of the output policy relies on a strong concentrability assumption, that makes it unsuitable for cases where the ratio between state-action distributions of behavior policy and some candidate policies is large. This is because in the traditional analysis, the error bound scales up with this ratio. We show that a small modification to Bellman optimality and evaluation b

[2007.08202] Provably Good Batch Reinforcement Learning Without Great Exploration Provably Good Batch Reinforcement Learning Without Great Exploration Yao Liu Stanford University yaoliu@stanford.edu &Adith Swaminathan Microsoft Research adswamin@microsoft.com Alekh Agarwal Microsoft Research alekha@microsoft.com &Emma Brunskill Stanford University ebrun@cs.stanford.edu Abstract Batch reinforcement learning (RL) is important to apply RL algorithms to many high stakes tasks. Doing batch RL in a way that yields a reliable new policy in large domains is challenging: a new decision policy may visit

related reading

• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• [1905.13341] On Value Functions and the Agent-Environment Boundaryar5iv.labs.arxiv.org
• [2307.04354] Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Dataar5iv.labs.arxiv.org
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• An Updated Introduction to Reinforcement Learning | Sri's Blogsrianumakonda.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• [2204.05618] When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?ar5iv.labs.arxiv.org
• Q-learning is not yet scalableseohong.me
• [2005.01643] Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemsar5iv.labs.arxiv.org
• Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
• [2211.05311] When is Realizability Sufficient for Off-Policy Reinforcement Learning?ar5iv.labs.arxiv.org