flâneur

What Matters for Latent Actions in Robot Learning

carldegio.github.io · 1,701 words · saved by 1 readers

Latent Action Models (LAMs) have emerged as a promising paradigm for enabling robot learning to leverage large-scale unlabeled videos through latent actions that serve as compact surrogates for physical actions. Despite rapid progress, research on LAM remains highly fragmented, with existing methods evaluating different design choices in isolation under inconsistent experimental settings, making it difficult to identify the factors that truly determine downstream robotic manipulation performance. In this work, we present the first comprehensive empirical study of latent action learning for robotic manipulation. We unify representative LAM methods within a common autoencoding framework and systematically investigate 41 LAM design choices across three dimensions, including latent action modeling paradigms, learning objectives and regularization methods, and latent action integration strategies. We further examine four proxy metrics for evaluating latent action quality and assess their ab

What Matters for Latent Actions in Robot Learning Xizhou Bu♦1, Qingda Hu♦1, Lei Zhou♦8, Lingfeng Zhang2, Yingbo Tang8, Zihao Liu4, Xinyi Tao3, Zhiqiang Ma6, Qingqiu Huang5, Chufeng Tang7, Hongbo Wang1, Jing Zhang5, Jiayi Ma5, Hangjun Ye8, Wei Li✻1, Xiaoshuai Hao✻8 ♦ Equal Contribution ✻ Corresponding Author 1 Fudan University 2 Tsinghua University 3 Sichuan University 4 Shenzhen University of Advanced Technology 5 Wuhan University 6 Suzhou Evans Intelligent Technology Co., Ltd. 7 Morphi Intelligence Technology Co., Ltd. 8 Xiaomi EV Project Page Arxiv Code Models Datasets Highlight…

saved by

related reading