IKEA-Manual: Seeing Shape Assembly Step by Step

Human-designed visual manuals are crucial components in shape assembly activities. They provide step-by-step guidance on how we should move and connect different parts in a convenient and physically-realizable way. While there has been an ongoing effort in building agents that perform assembly tasks, the information in human-design manuals has been largely overlooked. We identify that this is due to 1) a lack of realistic 3D assembly objects that have paired manuals and 2) the difficulty of extracting structured information from purely image-based manuals. Motivated by this observation, we present IKEA-Manual, a dataset consisting of 102 IKEA objects paired with assembly manuals. We provide fine-grained annotations on the IKEA objects and assembly manuals, including decomposed assembly parts, assembly plans, manual segmentation, and 2D-3D correspondence between 3D parts and visual manuals. We illustrate the broad application of our dataset on four tasks related to shape assembly: assem

Abstract Human-designed visual manuals are crucial components in shape assembly activities. They provide step-by-step guidance on how we should move and connect different parts in a convenient and physically-realizable way. While there has been an ongoing effort in building agents that perform assembly tasks, the information in human-design manuals has been largely overlooked. We identify that this is due to 1) a lack of realistic 3D assembly objects that have paired manuals and 2) the difficulty of extracting structured information from purely image-based manuals. Motivated by this…

saved by

• Breakfastbeforelunch

related reading

• The Building Blocks of Interpretabilitydistill.pub
• Generating Robot Hands from Human Demonstrationsarxiv.org
• Habitat-Matterport 3D Semantics Datasetopenaccess.thecvf.com
• ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoningh2t.iar.kit.edu
• 2D Amodal Instance Segmentation Guided by 3D Shape Priorecva.net
• Inferring Forces and Learning Human Utilities From Videoscv-foundation.org
• [2506.09985] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningarxiv.org
• [2409.17126] Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Resetarxiv.org
• Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingarxiv.org
• Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Modelsarxiv.org
• Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxesvirtualhumans.mpi-inf.mpg.de
• [2304.06718] Segment Everything Everywhere All at Oncearxiv.org