EfficientZero: How It Works - LessWrong

The goal of this essay is to help you understand EfficientZero, a reinforcement learning agent that obtains better-than-human median performance on a set of 26 Atari games after just two hours of real-time experience playing each game. Specifically, it gets 116% of human median performance on the data-limited Atari 100k benchmark. The previously-best algorithm only reached about 41% of median human performance, so this is a reasonably large leap. Chart stolen from paper The benchmark is called 100k because agents only interact with the environment for 100,000 steps -- about two hours. Note also that the human benchmarks were also set after the humans in question had about two hours of experience on the game. So EfficientZero seems to -- at least on this set of games -- exceed humans in sample efficiency specifically. This is particularly impressive when you recall that, going into this, the agents were entirely ignorant of anything whatsoever about the world. The networks of their brains were initialized with random values. Humans manage to do pretty well on Atari after two hours, but we've pretrained in the actual world for many years, which lets us apply analogies from this world, experience from other video games, and so on. This agent managed to do comparably well with none of these advantages. (Granted, the 100k benchmark focuses on Atari environments which are relatively easy to make progress in, because it was meant to be used for sample-efficiency benchmarks. It excludes extremely-difficult-to-explore environments like Montezuma's Revenge, where the first reward is quite hard to get.) So. After reading this, you should understand how EfficientZero works, and how the changes in EfficientZero improve upon its predecessors. You should also know what further improvements to it are likely in the coming 3 to 24 months. My target reader is someone who can program, or is at least broadly technically literate. They should be reasonably familiar with the bas

x EfficientZero: How It Works — LessWrong Best of LessWrong 2021 AI Capabilities Machine Learning (ML) Reinforcement learning EfficientZero AI Curated 307 EfficientZero: How It Works by 1a3orn 26th Nov 2021 AI Alignment Forum 35 min read 50 307 Ω 64 The goal of this essay is to help you understand EfficientZero , a reinforcement learning agent that obtains better-than-human median performance on a set of 26 Atari games after just two hours of real-time experience playing each game. Specifically, it gets 116% of human median performance on the data-limited Atari 100k benchmark. The previously-b

related reading

• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• AlphaGo Zero: Minimal Policy Improvement, Expectation Propagation and other Connectionsinference.vc
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• Evolution as Backstop for Reinforcement Learning · Gwern.netgwern.net
• Deep Q-Networks Explained — LessWronglesswrong.com
• Simple Alpha Zeroweb.stanford.edu
• pistar06.pdfpi.website
• Learning Beyond Gradientstrinkle23897.github.io
• [AN #70]: Agents that help humans who are still learning about their own preferences — LessWronglesswrong.com
• Reinforcement Learning in Newcomblike Problemsproceedings.neurips.cc
• How DeepMind's Generally Capable Agents Were Trained — LessWronglesswrong.com