A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence

Junyi Zhang1 Charles Herrmann2 Junhwa Hur2 Luisa F. Polanía2 Varun Jampani2 Deqing Sun2 Ming-Hsuan Yang2,3 1 Shanghai Jiao Tong University 2 Google Research 3 UC Merced NeurIPS 2023 Check out our follow-up work Telling Left from Right with better semantic correspondence! [Paper]      [Supp.]     [Arxiv]      [Code]     [BibTeX] Text-to-image diffusion models have made significant advances in generating and editing high-quality images. As a result, numerous approaches have explored the ability of diffusion model features to understand and process single images for downstream tasks, e.g., classification, semantic segmentation, and stylization. However, significantly less is known about what these features reveal across multiple, different images and objects. In this work, we exploit Stable Diffusion (SD) features for semantic and dense correspondence and discover that with simple post-processing, SD features can perform quantitatively similar to SOTA representations. Interestingly, t

A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence Junyi Zhang 1 Charles Herrmann 2 Junhwa Hur 2 Luisa F. Polanía 2 Varun Jampani 2 Deqing Sun 2 Ming-Hsuan Yang 2,3 1 Shanghai Jiao Tong University 2 Google Research 3 UC Merced NeurIPS 2023 Check out our follow-up work Telling Left from Right with better semantic correspondence! Semantic correspondence with fused Stable Diffusion and DINO features. --> On the left , we demonstrate the accuracy of our corre

saved by

• 3989

related reading

• Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencetelling-left-from-right.github.io
• What are Diffusion Models? | Lil'Loglilianweng.github.io
• The Illustrated Stable Diffusion – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• ⭐️ Diffusion Modelsandrewkchan.dev
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
• [2111.05826] Palette: Image-to-Image Diffusion Modelsarxiv.org
• How does Stable Diffusion work?stable-diffusion-art.com
• Yang Songyang-song.net
• 1703.10593arxiv.org
• Diffusion Meets Flow Matchingdiffusionflow.github.io
• 2311.15980arxiv.org
• Bare-bones Diffusion Modelsmadebyoll.in