Contrastive Representation Learning | Lil'Log

The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.

Table of Contents Contrastive Training Objectives Contrastive Loss Triplet Loss Lifted Structured Loss N-pair Loss NCE InfoNCE Soft-Nearest Neighbors Loss Common Setup Key Ingredients Heavy Data Augmentation Large Batch Size Hard Negative Mining Vision: Image Embedding Image Augmentations Basic Image Augmentation Augmentation Strategies Image Mixture Parallel Augmentation SimCLR Barlow Twins BYOL Memory Bank Instance Discrimination with Memoy Bank MoCo & MoCo-V2 CURL Feature Clustering DeepCluster SwAV Working with Supervised Datasets CLIP Supervised Contrastive Learning Language: Sentence Emb

related reading

• d89a66c7c80a29b1bdbab0f2a1a94af8-Paper.pdfproceedings.neurips.cc
• Untangling the Moons: A Visual History of Contrastive Learningtahabouhsine.com
• [2002.05709] A Simple Framework for Contrastive Learning of Visual Representationsarxiv.org
• Understanding self-supervised and contrastive learning with "Bootstrap Your Own Latent" (BYOL) - Imbueimbue.com
• Self-Supervised Representation Learning | Lil'Loglilianweng.github.io
• Contrastive Learning as Goal-Conditioned Reinforcement Learningarxiv.org
• The Annotated JEPA | Elements of a Vector Spaceelonlit.com
• Decoupling Representation Learning from Reinforcement Learningarxiv.org
• The Platonic Representation Hypothesisphillipi.github.io
• [2103.03230] Barlow Twins: Self-Supervised Learning via Redundancy Reductionarxiv.org
• Contrastive Code Representation Learningparasj.github.io
• Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Loglilianweng.github.io