Can Video World Models Track Unobserved World States?

An empirical study of hidden-state tracking in action-conditioned video world models, using a five-cup Shell Game as a visual analog of S5 state tracking.

Current video models render the visible world remarkably well, yet fail to track its unobserved state. Ground Truth Your browser does not support the video tag. Causal DiT (SWA1) · PSNR 40.4 / Acc 0.12 Causal DiT (SWA1) + LaCT · PSNR 41.1 / Acc 1.00 All models are trained with diffusion forcing on 5-swap episodes. At test time each model receives the clean frames of the opening reveal, then generates all 10 swaps and the final lift from the action sequence alone. Neither wider attention nor more denoising steps help. Proper state tracking requires a carried state and an expressive…

saved by

• Dhruv Sheth

related reading

• [2602.10098] VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelarxiv.org
• World Action Model Atlasjoeclinton.me
• [2404.02905] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictionarxiv.org
• Video models are zero-shot learners and reasonersarxiv.org
• Seoul World Model: Grounding World Simulation Models in a Real-World Metropolisseoul-world-model.github.io
• Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
• [2210.13382] Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskarxiv.org
• World Models | Rohit Bandarurohitbandaru.github.io
• How not to do research - Rajan Agarwalrajan.sh
• Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskarxiv.org
• [2603.08546] Interactive World Simulator for Robot Policy Training and Evaluationarxiv.org
• [2506.09985] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningarxiv.org