[2312.01853] Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing

Abstract:Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia. This approach allows for the simultaneous and seamless integration of both sensory inputs, offering richer spatial information and facilitating better reasoning about robot actions. The method, trained in a simulated environment and then deployed to a real robot, is applicable to various in-hand object rotation tasks. Comprehensive ablations are performed on how the integration of vision and touch can improve reinforcement learning and Sim2Real performance. Our project page is available at this https URL .

View PDF HTML (experimental) Abstract:Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloud-based tactile representation inspired by human tactile-visual synesthesia. This approach allows for the simultaneous and seamless integration of both sensory inputs, offering richer spatial…

saved by

• Jackie Yi

related reading

• Differentiable Robot Renderingdrrobot.cs.columbia.edu
• [2606.19161] HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Visionarxiv.org
• Emergence of Human to Robot Transfer in Vision-Language-Action Modelspi.website
• Generating Robot Hands from Human Demonstrationsarxiv.org
• [2602.06001] Visuo-Tactile World Modelsarxiv.org
• Soft Robotic Link with Controllable Transparency for Vision-based Tactile and Proximity Sensingquan-luu.github.io
• RL with Self-Supervised 3Dyanjieze.com
• [1910.07113] Solving Rubik's Cube with a Robot Handarxiv.org
• [2109.01115] Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationarxiv.org
• OSMO: Open-Source Tactile Glove for Human-to-Robot Skill Transferarxiv.org
• Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learningarxiv.org
• Vision in Action: Learning Active Perception from Human Demonstrationsvision-in-action.github.io