CLIP Enrichment Circuits

We investigate late circuits on CLIP's vision side to understand how they glue abstract concepts together and build more general multimodal neurons. Sidney Hough † Stanford University Kevin Liu † Stanford University Jack Ryan † Stanford University Chelsea Voss OpenAI Not published yet. No DOI yet. We build off of the work done in Multimodal Neurons in Artificial Neural Networks, diving deeper into the features in hidden layers of the CLIP model. In the spirit of the circuits agenda, we looked at individual units and their connections to attempt to unravel a small part of the algorithm CLIP uses to identify images. For context, the CLIP model we studied (CLIP-RN50) has two parts: a ResNet vision model and a Transformer language model . We investigated the ResNet side, which has as its final layer a relu layer with 2048 channels that combines via pairwise addition the activations from a conv2d layer and the previous relu layer (the residual connection). We confirm significant similaritie

CLIP Enrichment Circuits We investigate late circuits on CLIP's vision side to understand how they glue abstract concepts together and build more general multimodal neurons. Context We build off of the work done in Multimodal Neurons in Artificial Neural Networks , diving deeper into the features in hidden layers of the CLIP model . In the spirit of the circuits agenda , we looked at individual units and their connections to attempt to unravel a small part of the algorithm CLIP uses to identify images. For context, the CLIP model we studied (CLIP-RN50 This is a smaller model than the RN50-4x v

related reading

• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• Toy Models of Superpositiontransformer-circuits.pub
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Zoom In: An Introduction to Circuitsdistill.pub
• Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org
• On the Biology of a Large Language Modeltransformer-circuits.pub
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Transformer Circuits Threadtransformer-circuits.pub
• Circuit Tracing in Vision–Language Models:Understanding the Internal Mechanisms of Multimodal Thinkingarxiv.org
• Feature Visualizationdistill.pub
• CLIP · Hugging Facehuggingface.co
• Multimodal Neurons in Artificial Neural Networksdistill.pub