Curiosity-driven Exploration by Self-supervised Prediction | HTML5

In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent’s ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, critically, ignores the aspects of the environment that cannot affect the agent. The proposed approach is evaluated in two environments: VizDoom and Super Mario Bros. Three broad settings are investigated: 1) sparse extrinsic reward, where curiosity allows for far fewer interactions with the environment to reach the goal; 2) exploration with no extrinsic reward, where curiosity pushes the agent to

Curiosity-driven Exploration by Self-supervised Prediction Deepak Pathak Pulkit Agrawal Alexei A. Efros Trevor Darrell Abstract In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent’s ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation

related reading

• Exploration Strategies in Deep Reinforcement Learning | Lil'Loglilianweng.github.io
• pdfopenreview.net
• [1606.01868] Unifying Count-Based Exploration and Intrinsic Motivationar5iv.labs.arxiv.org
• Reward is not the optimization target — LessWronglesswrong.com
• The Era of Experience Paper.pdfstorage.googleapis.com
• 0812.4360arxiv.org
• Intrinsically-Motivated Humans and Agents in Open-World Exploration | Aly Lidayanalyd.github.io
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• [2201.11861] The Challenges of Exploration for Offline Reinforcement Learningar5iv.labs.arxiv.org
• [2210.05805] Exploration via Elliptical Episodic Bonusesar5iv.labs.arxiv.org
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• How DeepMind's Generally Capable Agents Were Trained — LessWronglesswrong.com