[2301.13823] Grounding Language Models to Images for Multimodal Generation

We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process and generate arbitrarily interleaved image-and-text data. Our method leverages the abilities of language models learnt from large scale text-only pretraining, such as in-context learning and free-form text generation. We keep the language model frozen, and finetune input and output linear layers to enable cross-modality interactions. This allows our model to process arbitrarily interleaved image-and-text inputs, and generate free-form text interleaved with retrieved images. We achieve strong zero-shot performance on grounded tasks such as contextual image retrieval and multimodal dialogue, and showcase compelling interactive abilities. Our approach works with any off-the-shelf language model and paves the way towards an effective, general solution for leveraging pretrained language models in visually grounded settings.

Grounding Language Models to Images for Multimodal Inputs and Outputs Jing Yu Koh 1 Ruslan Salakhutdinov 1 Daniel Fried 1 Abstract arXiv:2301.13823v4 [cs.CL] 13 Jun 2023 We propose an efficient method to ground pre- trained text-only language models to the visual domain, enabling them to process…

saved by

• Kalie C

related reading

• Multi-View Transformer for 3D Visual Groundingarxiv.org
• 2403.09611.pdfarxiv.org
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• [2309.10668] Language Modeling Is Compressionarxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• Unified Multimodal Models as Auto-Encodersarxiv.org
• [2210.13382] Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskarxiv.org
• Composing Zero-Shot Multimodal Reasoning with Languagesocraticmodels.github.io
• 2306.17806arxiv.org
• Language Models can Solve Computer Tasksarxiv.org
• [1912.13318] LayoutLM: Pre-training of Text and Layout for Document Image Understandingarxiv.org
• [2509.14252] LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturesarxiv.org