[2606.02800] Cosmos 3: Omnimodal World Models for Physical AI

Abstract:We introduce Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-transformers architecture. By supporting highly flexible input-output configurations, Cosmos 3 seamlessly unifies critical modalities for Physical AI -- effectively subsuming vision-language models, video generators, world simulators, and world-action models into a single framework. Our evaluation demonstrates that Cosmos 3 establishes a new state-of-the-art across a diverse suite of understanding and generation tasks, demonstrating omnimodal world models as scalable, general-purpose backbones for embodied agents. Our post-trained Cosmos 3 models were ranked as the best open-source Text-to-Image and Image-to-Video models by Artificial Analysis, and the best policy model by RoboArena at the time the technical report was written. To accelerate open research and deployment in Physical AI, we make our code, model checkpoints, curated synthetic datasets, and evaluation benchmark available under the Linux Foundation's OpenMDW-1.1 License at this https URL and this https URL. The project website is available at this https URL.

# link_1ag7y2ccllr.pdf ## Metadata - PDFFormatVersion=1.7 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=NVIDIA; :; Aditi; Niket Agarwal; Arslan Ali; Jon Allen; Martin Antolini; Adeline Aubame; Alisson Azzolini; Junjie Bai; Maciej Bala; Yogesh Balaji; Josh Bapst; Aarti Basant; Mukesh Beladiya; Mohammad Qazim Bhat; Zaid Pervaiz Bhat; Dan Blick; Vanni Brighella; Han Cai; Tiffany Cai; Eric Cameracci; Jiaxin Cao; Yulong Cao; Mark Carlson; Carlos Casanova; Ting-Yun Chang; Yan Chang; Yu-Wei Chao; Prithvijit Chattop

saved by

• John Zhang

related reading

• A Unified Model for Motion-Conditioned Robot Co-designtransformer-transformer.github.io
• pdfopenreview.net
• Generalist - GEN-0 / Embodied Foundation Models That Scale with Physical Interactiongeneralistai.com
• [2603.08546] Interactive World Simulator for Robot Policy Training and Evaluationarxiv.org
• Video models are zero-shot learners and reasonersarxiv.org
• 2403.09611.pdfarxiv.org
• [2603.05438] Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Modelarxiv.org
• [2603.16666] Fast-WAM: Do World Action Models Need Test-time Future Imagination?arxiv.org
• Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models | NVIDIA Technical Blogdeveloper.nvidia.com
• World Models | Rohit Bandarurohitbandaru.github.io
• Efficient World Models with Context-Aware Tokenizationarxiv.org
• [2506.09985] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningarxiv.org