CLIP

The CLIP model was proposed in Learning Transferable Visual Models From Natural Language Supervision by Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. The abstract from the paper is the following: State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much

This model was published in HF papers on 2021-02-26 and contributed to Hugging Face Transformers on 2021-05-12. CLIP is a multimodal vision and language model motivated by overcoming the fixed number of object categories when training a computer vision model. CLIP learns about images directly from raw text by jointly training on 400M (image, text) pairs. Pretraining on this scale enables zero-shot transfer to downstream tasks. CLIP uses an image encoder and text encoder to get visual features and text features. Both features are projected to a latent space with the same number of dimensions…

related reading

• Replicate - Run AI with an APIreplicate.com
• Hugging Face – The AI community building the future.huggingface.co
• [2103.00020] Learning Transferable Visual Models From Natural Language Supervisionarxiv.org
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• 2403.09611.pdfarxiv.org
• Llama 2 · Hugging Facehuggingface.co
• [2301.13823] Grounding Language Models to Images for Multimodal Generationarxiv.org
• Adversarial examples for the OpenAI CLIP in its zero-shot classification regime and their semantic generalization | Stanislav Fortstanislavfort.github.io
• Feature-wise transformationsdistill.pub
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelsarxiv.org
• 2310.12921.pdfarxiv.org
• Annotated Research Paper Implementations: Transformers, StyleGAN, Stable Diffusion, DDPM/DDIM, LayerNorm, Nucleus Sampling and morenn.labml.ai