Temporal Difference Models: Model-Free Deep RL for Model-Based Control | HTML5

Model-free reinforcement learning (RL) is a powerful, general tool for learning complex behaviors. However, its sample efficiency is often impractically large for solving challenging real-world problems, even with off-policy algorithms such as Q-learning. A limiting factor in classic model-free RL is that the learning signal consists only of scalar rewards, ignoring much of the rich information contained in state transition tuples. Model-based RL uses this information, by training a predictive model, but often does not achieve the same asymptotic performance as model-free RL due to model bias. We introduce temporal difference models (TDMs), a family of goal-conditioned value functions that can be trained with model-free learning and used for model-based control. TDMs combine the benefits of model-free and model-based RL: they leverage the rich information in state transitions to learn very efficiently, while still attaining asymptotic performance that exceeds that of direct model-based

Temporal Difference Models: Model-Free Deep RL for Model-Based Control Vitchyr Pong University of California, Berkeley vitchyr@berkeley.edu &Shixiang Gu 1 1 footnotemark: 1 University of Cambridge Max Planck Institute Google Brain sg717@cam.ac.uk \AND Murtaza Dalal University of California, Berkeley mdalal@berkeley.edu &Sergey Levine University of California, Berkeley svlevine@eecs.berkeley.edu denotes equal contribution Abstract Model-free reinforcement learning (RL) is a powerful, general tool for learning complex behaviors. However, its sample efficiency is often impractically large for sol

related reading

• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Part 2: Kinds of RL Algorithms - Spinning Up documentationspinningup.openai.com
• Model-free (reinforcement learning) - Wikipediaen.wikipedia.org
• Key Papers in Deep RL - Spinning Up documentationspinningup.openai.com
• Q-learning is not yet scalableseohong.me
• [1805.12114] Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelsar5iv.labs.arxiv.org
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• [2006.03647] Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationar5iv.labs.arxiv.org
• Introducing Dreamer: Scalable Reinforcement Learning Using World Modelsresearch.google
• Deep Q-Networks Explained — LessWronglesswrong.com
• The Promise of Hierarchical Reinforcement Learningthegradient.pub
• Reinforcement learning - Wikipediaen.wikipedia.org