[2509.02722] Planning with Reasoning using Vision Language World Model

Abstract:Effective planning requires strong world models, but high-level world models that can understand and reason about actions with semantic and temporal abstraction remain largely underdeveloped. We introduce the Vision Language World Model (VLWM), a foundation model trained for language-based world modeling on natural videos. Given visual observations, the VLWM first infers the overall goal achievements then predicts a trajectory composed of interleaved actions and world state changes. Those targets are extracted by iterative LLM Self-Refine conditioned on compressed future observations represented by Tree of Captions. The VLWM learns both an action policy and a dynamics model, which respectively facilitates reactive system-1 plan decoding and reflective system-2 planning via cost minimization. The cost evaluates the semantic distance between the hypothetical future states given by VLWM roll-outs and the expected goal state, and is measured by a critic model that we trained in a self-supervised manner. The VLWM achieves state-of-the-art Visual Planning for Assistance (VPA) performance on both benchmark evaluations and our proposed PlannerArena human evaluations, where system-2 improves the Elo score by +27% upon system-1. The VLWM models also outperforms strong VLM baselines on RoboVQA and WorldPrediction benchmark.

[2509.02722] Planning with Reasoning using Vision Language World Model --> Computer Science > Artificial Intelligence arXiv:2509.02722 (cs) [Submitted on 2 Sep 2025 ( v1 ), last revised 6 Sep 2025 (this version, v2)] Title: Planning with Reasoning using Vision Language World Model Authors: Delong Chen , Theo Moutakanni , Willy Chung , Yejin Bang , Ziwei Ji , Allen Bolourchi , Pascale Fung View a PDF of the paper titled Planning with Reasoning using Vision Language World Model, by Delong Chen and 6 other authors View PDF HTML (experimental) Abstract: Effective planning requires strong world mod

saved by

• Sofia Bodnar

related reading

• [2506.09985] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningarxiv.org
• Video Language Planningvideo-language-planning.github.io
• [2201.07207] Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentsarxiv.org
• Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models | NVIDIA Technical Blogdeveloper.nvidia.com
• [2603.16666] Fast-WAM: Do World Action Models Need Test-time Future Imagination?arxiv.org
• RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackarxiv.org
• Video models are zero-shot learners and reasonersarxiv.org
• [2504.15785] WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agentsarxiv.org
• [2603.21972] Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipearxiv.org
• [2602.10556] LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transferarxiv.org
• [2603.05923] Learning Next Action Predictors from Human-Computer Interactionarxiv.org
• RT-2: Vision-Language-Action Modelsrobotics-transformer2.github.io