Untangling the Moons: A Visual History of Contrastive Learning

Eight contrastive losses, twenty years of history, one interactive playground. Watch pair, triplet, InfoNCE, CLIP, SupCon, SigLIP, alignment+uniformity, and cosine→0 organize 2D points — and see which ones know when to stop.

⌨ Runnable JAX companion Organizing Randomness: Contrastive Learning in JAX Prefer to read the code? This post has a hands-on JAX / Flax NNX implementation. Open the JAX companion → Twenty years of contrastive learning, eight losses, eight datasets, and the question of when a loss should stop pushing. Contrastive learning is the standard recipe for turning raw data into a usable embedding space, and the recipe has been rewritten roughly every three years. As a framing device, and it is a framing device, the real history is messier, each one reads as a response to the previous one’s failure mod

saved by

• Sean Lee

related reading

• [2410.21265] Modular Duality in Deep Learningarxiv.org
• d89a66c7c80a29b1bdbab0f2a1a94af8-Paper.pdfproceedings.neurips.cc
• Feature Visualizationdistill.pub
• NL.pdfabehrouz.github.io
• The Annotated JEPA | Elements of a Vector Spaceelonlit.com
• Contrastive Representation Learning | Lil'Loglilianweng.github.io
• The Continual Learning Problemjessylin.com
• [2002.05709] A Simple Framework for Contrastive Learning of Visual Representationsarxiv.org
• Understanding Convolutions on Graphsdistill.pub
• CS231n Deep Learning for Computer Visioncs231n.github.io
• What is Contrastive Learning? (Contrastive Learning/Self-supervised Learning Explained)youtube.com
• Tensor-Transformer Variants are Surprisingly Performant — LessWronglesswrong.com