[2006.07733] Bootstrap your own latent: A new approach to self-supervised Learning

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.

[2006.07733] Bootstrap your own latent: A new approach to self-supervised Learning Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2006.07733 (cs) [Submitted on 13 Jun 2020 ( v1 ), last revised 10 Sep 2020 (this version, v3)] Title: Bootstrap your own latent: A new approach to self-supervised Learning Authors: Jean-Bastien Grill , Florian Strub , Florent Altché , Corentin Tallec , Pierre H. Richemond , Elena Buchatskaya , Carl Doersch , Bernardo Avila Pires ,

related reading

• Understanding self-supervised and contrastive learning with "Bootstrap Your Own Latent" (BYOL) - Imbueimbue.com
• Self-supervised learning - Wikipediaen.wikipedia.org
• Self-Supervised Representation Learning | Lil'Loglilianweng.github.io
• [2002.05709] A Simple Framework for Contrastive Learning of Visual Representationsarxiv.org
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Loglilianweng.github.io
• Unsupervised Learning with SURE.pdfarxiv.org
• A Few Words on Representation Learning - Thalles' blogsthalles.github.io
• On the Stepwise Nature of Self-Supervised Learning - Imbuegenerallyintelligent.com
• Contrastive Representation Learning | Lil'Loglilianweng.github.io
• Neighbor2Neighborarxiv.org
• Learning to be Bayesian without Supervisionpapers.nips.cc