flâneur

[2510.13626] LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models

arxiv.org · 6,954 words · saved by 1 readers

Abstract:Visual-Language-Action (VLA) models report impressive success rates on robotic manipulation benchmarks, yet these results may mask fundamental weaknesses in robustness. We perform a systematic vulnerability analysis by introducing controlled perturbations across seven dimensions: objects layout, camera viewpoints, robot initial states, language instructions, light conditions, background textures and sensor noise. We comprehensively analyzed multiple state-of-the-art models and revealed consistent brittleness beneath apparent competence. Our analysis exposes critical weaknesses: models exhibit extreme sensitivity to perturbation factors, including camera viewpoints and robot initial states, with performance dropping from 95% to below 30% under modest perturbations. Surprisingly, models are largely insensitive to language variations, with further experiments revealing that models tend to ignore language instructions completely. Our findings challenge the assumption that high benchmark scores equate to true competency and highlight the need for evaluation practices that assess reliability under realistic variation.

OpenMOSS LIBERO-Plus: In-depth Robustness Analysis of Vision-Language- Action Models Senyu Fei2,3,† Siyin Wang1,3,†,∗ Junhao Shi1,3,† Zihao Dai1,‡ Jikun Cai1,‡ Pengfang Qian1,3,‡ Li Ji1 Xinzhe He1 Shiduo Zhang1 Zhaoye Fei1 Jinlan Fu4 Jingjing Gong3,B Xipeng Qiu1,3,B 1 Fudan University 2 Tongji University 3 Shanghai Innovation…

saved by

related reading