Contrastive Representation Learning | Lil'Log

The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.

Table of Contents Contrastive Training Objectives Contrastive Loss Triplet Loss Lifted Structured Loss N-pair Loss NCE InfoNCE Soft-Nearest Neighbors Loss Common Setup Key Ingredients Heavy Data Augmentation Large Batch Size Hard Negative Mining Vision: Image Embedding Image Augmentations Basic Image Augmentation Augmentation Strategies Image Mixture Parallel Augmentation SimCLR Barlow Twins BYOL Memory Bank Instance Discrimination with Memoy Bank MoCo & MoCo-V2 CURL Feature Clustering DeepCluster SwAV Working with Supervised Datasets CLIP Supervised Contrastive Learning Language: Sentence Emb

related reading

• Untangling the Moons: A Visual History of Contrastive Learningtahabouhsine.com
• Understanding self-supervised and contrastive learning with "Bootstrap Your Own Latent" (BYOL) - Imbueimbue.com
• Self-Supervised Representation Learning | Lil'Loglilianweng.github.io
• [2002.05709] A Simple Framework for Contrastive Learning of Visual Representationsarxiv.org
• The Annotated JEPA | Elements of a Vector Spaceelonlit.com
• The Platonic Representation Hypothesisphillipi.github.io
• Contrastive Code Representation Learningparasj.github.io
• Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Loglilianweng.github.io
• Generative modelling in latent space – Sander Dielemansander.ai
• The Illustrated Word2vec – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Few Words on Representation Learning - Thalles' blogsthalles.github.io
• Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorsmedarc-ai.github.io