✳flâneur — a map of the web's best reading

Semantic World Models

weirdlabuw.github.io · 1,132 words · saved by 1 readers

Planning with world models offers a powerful paradigm for robotic control. Conventional approaches train a model to predict future frames conditioned on current frames and actions, which can then be used for planning. However, the objective of predicting future pixels is often at odds with the actual planning objective; strong pixel reconstruction does not always correlate with good planning decisions. We posit that instead of reconstructing future frames as pixels, world models only need to predict task-relevant semantic information about the future. To do this, we pose world modeling as a visual question answering problem, about semantic information in future frames. This perspective allows world modeling to be approached with the same tools underlying vision language models. We show how vision language models can be trained as "semantic world models" through a supervised finetuning process on image-action-text data, enabling planning for decision-making while inheriting many of the

Semantic World Models Semantic World Models Jacob Berg 1 , Chuning Zhu 1 , Yanda Bao 1 , Ishan Durugkar 2 , Abhishek Gupta 1 1 University of Washington 2 Sony AI Paper arXiv Video --> Demo Abstract tl;dr Learning world models using visual question answering about the future . --> Planning with world models offers a powerful paradigm for robotic control. Conventional approaches train a model to predict future frames conditioned on current frames and actions, which can then be used for planning. However, the objective of predicting future pixels is often at odds with the actual planning objectiv

Explore this link on the map →

related reading