Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Large pretrained (e.g., “foundation”) models exhibit distinct capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language models (VLMs) are trained on Internet-scale image captions, but large language models (LMs) are further trained on Internet-scale text with no images (e.g., spreadsheets, SAT questions, code). As a result, these models store different forms of commonsense knowledge across different domains. In this work, we show that this diversity is symbiotic, and can be leveraged through Socratic Models (SMs): a modular framework in which multiple pretrained models may be composed zero-shot i.e., via multimodal-informed prompting, to exchange information with each other and capture new multimodal capabilities, without requiring finetuning. With minimal engineering, SMs are not only competitive with state-of-the-art zero-shot image captioning and video-to-text retrieval, but also enab

Socratic Models Composing Zero-Shot Multimodal Reasoning with Language Paper Code Abstract Large pretrained (e.g., “foundation”) models exhibit distinct capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language models (VLMs) are trained on Internet-scale image captions, but large language models (LMs) are further trained on Internet-scale text with no images (e.g., spreadsheets, SAT questions, code). As a result, these models store different forms of commonsense knowledge across different…

related reading

• [2301.13823] Grounding Language Models to Images for Multimodal Generationarxiv.org
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• Video models are zero-shot learners and reasonersarxiv.org
• 2403.09611.pdfarxiv.org
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• [2205.11916] Large Language Models are Zero-Shot Reasonersarxiv.org
• [2201.07207] Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentsarxiv.org
• Language Model Compositionarxiv.org
• Language Models can Solve Computer Tasksarxiv.org
• Circuit Tracing in Vision–Language Models:Understanding the Internal Mechanisms of Multimodal Thinkingarxiv.org