Bootstrapped Meta-Learning

Meta-learning empowers artificial intelligence to increase its efficiency by learning how to learn. Unlocking this potential involves overcoming a challenging meta-optimisation problem that often exhibits ill-conditioning, and myopic meta-objectives. We propose an algorithm that tackles these issues by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-)metric. Focusing on meta-learning with gradients, we establish conditions that guarantee performance improvements and show that the improvement is related to the target distance. Thus, by controlling curvature, the distance measure can be used to ease meta-optimization, for instance by reducing ill-conditioning. Further, the bootstrapping mechanism can extend the effective meta-learning horizon without requiring backpropagation through all updates. The algorithm is versatile and easy to implement. We achieve a new state-of-the art for model-free agents on the Atari ALE benchmark, improve upon MAML in few-shot learning, and demonstrate how our approach opens up new possibilities by meta-learning efficient exploration in a Q-learning agent.

Published as a conference paper at ICLR 2022 B OOTSTRAPPED M ETA -L EARNING Sebastian Flennerhag Yannick Schroecker Tom Zahavy DeepMind DeepMind DeepMind flennerhag@google.com Hado van Hasselt David Silver Satinder Singh…

related reading

• [1906.01820] Risks from Learned Optimization in Advanced Machine Learning Systemsarxiv.org
• Meta-Learning: Learning to Learn Fastlilianweng.github.io
• NL.pdfabehrouz.github.io
• Metalearning or Learning to Learn Since 1987people.idsia.ch
• Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• Risks from Learned Optimization: Introduction — LessWronglesswrong.com
• [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• [2602.11399] Can We Really Learn One Representation to Optimize All Rewards?arxiv.org
• [2602.17616] Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMsarxiv.org
• [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org