Exploration Strategies in Deep Reinforcement Learning | Lil'Log

Exploitation versus exploration is a critical topic in Reinforcement Learning. We’d like the RL agent to find the best solution as fast as possible. However, in the meantime, committing to solutions too quickly without enough exploration sounds pretty bad, as it could lead to local minima or total failure. Modern RL algorithms that optimize for the best returns can achieve good exploitation quite efficiently, while exploration remains more like an open topic. I would like to discuss several common exploration strategies in Deep RL here. As this is a very big topic, my post by no means can cover all the important subtopics. I plan to update it periodically and keep further enriching the content gradually in time. As a quick recap, let’s first go through several classic exploration algorithms that work out pretty well in the multi-armed bandit problem or simple tabular RL. The following strategies could be used for better exploration in deep RL training when neural networks are used for

Table of Contents Classic Exploration Strategies Key Exploration Problems The Hard-Exploration Problem The Noisy-TV Problem Intrinsic Rewards as Exploration Bonuses Count-based Exploration Counting by Density Model Counting after Hashing Prediction-based Exploration Forward Dynamics Random Networks Physical Properties Memory-based Exploration Episodic Memory Direct Exploration Q-Value Exploration Varitional Options Citation Reference [Updated on 2020-06-17: Add "exploration via disagreement" in the "Forward Dynamics" section . Exploitation versus exploration is a critical topic in Reinforcemen

related reading

• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• Debugging Reinforcement Learning Systemsandyljones.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Part 2: Kinds of RL Algorithms - Spinning Up documentationspinningup.openai.com
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• Key Papers in Deep RL - Spinning Up documentationspinningup.openai.com
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• The Multi-Armed Bandit Problem and Its Solutions | Lil'Loglilianweng.github.io
• Deep Q-Networks Explained — LessWronglesswrong.com
• How to Explore to Scale RL Training of LLMs on Hard Problems? – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu