Contrastive Representation Learning | Lil'Log

The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.

Table of Contents Contrastive Training Objectives Contrastive Loss Triplet Loss Lifted Structured Loss N-pair Loss NCE InfoNCE Soft-Nearest Neighbors Loss Common Setup Key Ingredients Heavy Data Augmentation Large Batch Size Hard Negative Mining Vision: Image Embedding Image Augmentations Basic Image Augmentation Augmentation Strategies Image Mixture Parallel Augmentation SimCLR Barlow Twins BYOL Memory Bank Instance Discrimination with Memoy Bank MoCo & MoCo-V2 CURL Feature Clustering DeepCluster SwAV Working with Supervised Datasets CLIP Supervised Contrastive Learning Language: Sentence Emb

related reading

• Untangling the Moons: A Visual History of Contrastive Learningtahabouhsine.com
• Understanding self-supervised and contrastive learning with "Bootstrap Your Own Latent" (BYOL) - Imbueimbue.com
• Self-Supervised Representation Learning | Lil'Loglilianweng.github.io
• [2002.05709] A Simple Framework for Contrastive Learning of Visual Representationsarxiv.org
• The Platonic Representation Hypothesisphillipi.github.io
• Contrastive Code Representation Learningparasj.github.io
• Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Loglilianweng.github.io
• Generative modelling in latent space – Sander Dielemansander.ai
• The Annotated JEPA | Elements of a Vector Spaceelonlit.com
• A Few Words on Representation Learning - Thalles' blogsthalles.github.io
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Contrastive Representations for Temporal Reasoningarxiv.org