Semantic World Models

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

Semantic World Models Jacob Berg 1 Chuning Zhu 1 Yanda Bao 1 Ishan Durugkar 2 Abhishek Gupta 1 1 University of Washington 2 Sony AI jacob33,zchuning,yandabao,abhgupta@cs.washington.edu ishan.durugkar@sony.com Abstract Planning with world models offers a powerful paradigm for robotic control. Conventional approaches train a model to predict future frames conditioned on current frames and actions, which can then be used for planning. However, the objective of predicting future pixels is often at odds with the actual planning objective; strong pixel reconstruction does not always correlate with g

related reading

• Semantic World Modelsweirdlabuw.github.io
• World Models | Rohit Bandarurohitbandaru.github.io
• pdfopenreview.net
• Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models | NVIDIA Technical Blogdeveloper.nvidia.com
• [2603.08546] Interactive World Simulator for Robot Policy Training and Evaluationarxiv.org
• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelsle-wm.github.io
• [2509.02722] Planning with Reasoning using Vision Language World Modelarxiv.org
• [2506.09985] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningarxiv.org
• [2602.06001] Visuo-Tactile World Modelsarxiv.org
• [2603.16666] Fast-WAM: Do World Action Models Need Test-time Future Imagination?arxiv.org
• Efficient World Models with Context-Aware Tokenizationarxiv.org
• RT-2: Vision-Language-Action Modelsrobotics-transformer2.github.io