Bootstrapped Meta-Learning

Meta-learning empowers artificial intelligence to increase its efficiency by learning how to learn. Unlocking this potential involves overcoming a challenging meta-optimisation problem that often exhibits ill-conditioning, and myopic meta-objectives. We propose an algorithm that tackles these issues by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-)metric. Focusing on meta-learning with gradients, we establish conditions that guarantee performance improvements and show that the improvement is related to the target distance. Thus, by controlling curvature, the distance measure can be used to ease meta-optimization, for instance by reducing ill-conditioning. Further, the bootstrapping mechanism can extend the effective meta-learning horizon without requiring backpropagation through all updates. The algorithm is versatile and easy to implement. We achieve a new state-of-the art for model-free agents on the Atari ALE benchmark, improve upon MAML in few-shot learning, and demonstrate how our approach opens up new possibilities by meta-learning efficient exploration in a Q-learning agent.

Published as a conference paper at ICLR 2022 B OOTSTRAPPED M ETA -L EARNING Sebastian Flennerhag Yannick Schroecker Tom Zahavy DeepMind DeepMind DeepMind flennerhag@google.com Hado van Hasselt David Silver Satinder Singh…

related reading

• Meta-Learning: Learning to Learn Fastlilianweng.github.io
• Metalearning or Learning to Learn Since 1987people.idsia.ch
• Just Ask for Generalization | Eric Jangevjang.com
• why don’t we explicitly train models to be good at generalization?substack.com
• NL.pdfabehrouz.github.io
• [1703.03400] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networksarxiv.org
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Evolution as Backstop for Reinforcement Learning · Gwern.netgwern.net
• [2301.08028] A Tutorial on Meta-Reinforcement Learningarxiv.org
• [2212.07677] Transformers learn in-context by gradient descentarxiv.org
• Learning Beyond Gradientstrinkle23897.github.io
• Why We Need Continual Learning | Andreessen Horowitza16z.com