Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning | Meta AI Research

We present CM3Leon (pronounced “Chameleon”), a retrieval-augmented, tokenbased, decoder-only multi-modal language model capable of generating and...

NLP COMPUTER VISION July 14, 2023 Abstract We present CM3Leon (pronounced “Chameleon”), a retrieval-augmented, tokenbased, decoder-only multi-modal language model capable of generating and infilling both text and images. CM3Leon uses the CM3 multi-modal architecture but additionally shows the extreme benefits of scaling up and tuning on more diverse instruction-style data. It is the first multi-modal model trained with a recipe adapted from text-only language models, including a large-scale retrieval-augmented pretraining stage and a second multi-task supervised fine-tuning (SFT) stage.…

saved by

• Justine Massicotte

related reading

• Introducing CM3leon, a more efficient, state-of-the-art generative model for text and imagesai.meta.com
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• 2403.09611.pdfarxiv.org
• [2301.13823] Grounding Language Models to Images for Multimodal Generationarxiv.org
• [2512.11949] Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitorsarxiv.org
• Large Language Diffusion Modelsarxiv.org
• Self-Adapting Language Modelsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Unified Multimodal Models as Auto-Encodersarxiv.org
• Inside a Neural Chameleonjacksonmowattgok.com
• Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Modelsarxiv.org
• Language Model Compositionarxiv.org