✳flâneur — a map of the web's best reading

Video Language Planning

video-language-planning.github.io · 2,091 words · saved by 1 readers

We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data. To this end, we present video language planning (VLP), an algorithm that consists of a tree search procedure, where we train (i) vision-language models to serve as both policies and value functions, and (ii) text-to-video models as dynamics models. VLP takes as input a long-horizon task instruction and current image observation, and outputs a long video plan that provides detailed multimodal (video and language) specifications that describe how to complete the final task. VLP scales with increasing computation budget where more computation time results in improved video plans, and is able to synthesize long-horizon video plans across different robotics domains – from multi-object rearrangement, to multi-camera bi-arm dexterous manipulation. Generated video plans can be transla

Video Language Planning Video Language Planning Yilun Du Sherry Yang Pete Florence Fei Xia Ayzaan Wahid Brian Ichter Pierre Sermanet Tianhe Yu Pieter Abbeel Joshua B. Tenenbaum Leslie Kaelbling Andy Zeng Jonathan Tompson Google DeepMind MIT UC Berkeley Paper Code Data Results Abstract We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data. To this end, we present video language planning (VLP), an algorithm that consists of a tree search pr

Explore this link on the map →

saved by

related reading