Exploration Strategies in Deep Reinforcement Learning | Lil'Log

Exploitation versus exploration is a critical topic in Reinforcement Learning. We’d like the RL agent to find the best solution as fast as possible. However, in the meantime, committing to solutions too quickly without enough exploration sounds pretty bad, as it could lead to local minima or total failure. Modern RL algorithms that optimize for the best returns can achieve good exploitation quite efficiently, while exploration remains more like an open topic. I would like to discuss several common exploration strategies in Deep RL here. As this is a very big topic, my post by no means can cover all the important subtopics. I plan to update it periodically and keep further enriching the content gradually in time. As a quick recap, let’s first go through several classic exploration algorithms that work out pretty well in the multi-armed bandit problem or simple tabular RL. The following strategies could be used for better exploration in deep RL training when neural networks are used for

Table of Contents Classic Exploration Strategies Key Exploration Problems The Hard-Exploration Problem The Noisy-TV Problem Intrinsic Rewards as Exploration Bonuses Count-based Exploration Counting by Density Model Counting after Hashing Prediction-based Exploration Forward Dynamics Random Networks Physical Properties Memory-based Exploration Episodic Memory Direct Exploration Q-Value Exploration Varitional Options Citation Reference [Updated on 2020-06-17: Add "exploration via disagreement" in the "Forward Dynamics" section . Exploitation versus exploration is a critical topic in Reinforcemen

related reading

• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Key Papers in Deep RL - Spinning Up documentationspinningup.openai.com
• [2210.05805] Exploration via Elliptical Episodic Bonusesar5iv.labs.arxiv.org
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• [1606.01868] Unifying Count-Based Exploration and Intrinsic Motivationar5iv.labs.arxiv.org
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• [1705.05363] Curiosity-driven Exploration by Self-supervised Predictionar5iv.labs.arxiv.org
• Reinforcement Learning via Implicit Imitation Guidancearxiv.org
• Reward is not the optimization target — LessWronglesswrong.com
• Deep Q-Networks Explained — LessWronglesswrong.com
• The Multi-Armed Bandit Problem and Its Solutions | Lil'Loglilianweng.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de