Language-Grounded Indoor 3D Semantic Segmentation in the Wild

Recent advances in 3D semantic segmentation with deep neural networks have shown remarkable success, with rapid performance increase on available datasets. However, current 3D semantic segmentation benchmarks contain only a small number of categories -- less than 30 for ScanNet and SemanticKITTI, for instance, which are not enough to reflect the diversity of real environments (e.g., semantic image understanding covers hundreds to thousands of classes). Thus, we propose to study a larger vocabulary for 3D semantic segmentation with a new extended benchmark on ScanNet data with 200 class categories, an order of magnitude more than previously studied. This large number of class categories also induces a large natural class imbalance, both of which are challenging for existing 3D semantic segmentation methods. To learn more robust 3D features in this context, we propose a language-driven pre-training method to encourage learned 3D features that might have limited training examples to lie close to their pre-trained text embeddings. Extensive experiments show that our approach consistently outperforms state-of-the-art 3D pre-training for 3D semantic segmentation on our proposed benchmark (+9% relative mIoU), including limited-data scenarios with +25% relative mIoU using only 5% annotations.

Recent advances in 3D semantic segmentation with deep neural networks have shown remarkable success, with rapid performance increase on available datasets. However, current 3D semantic segmentation benchmarks contain only a small number of categories -- less than 30 for ScanNet and SemanticKITTI, for instance, which are not enough to reflect the diversity of real environments (e.g., semantic image understanding covers hundreds to thousands of classes). Thus, we propose to study a larger vocabulary for 3D semantic segmentation with a new extended benchmark on ScanNet data with 200 class categor

saved by

• Kalie C

related reading

• PLA: Language-Driven Open-Vocabulary 3D Scene Understandingarxiv.org
• ReferIt3D Benchmarksreferit3d.github.io
• ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenesecva.net
• Multi-View Transformer for 3D Visual Groundingarxiv.org
• [2310.08586] PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmarxiv.org
• [2206.10555] LargeKernel3D: Scaling up Kernels in 3D Sparse CNNsarxiv.org
• Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencetelling-left-from-right.github.io
• 3DMV3dmv2023.github.io
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• 3D-LFM: Lifting Foundation Model3dlfm.github.io
• [2212.02493] Canonical Fields: Self-Supervised Learning of Pose-Canonicalized Neural Fieldsarxiv.org
• Universal consensus 3D segmentation of cells from 2D segmented stacks | Nature Methodsnature.com