[2211.10440] Magic3D: High-Resolution Text-to-3D Content Creation

DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields (NeRF), achieving remarkable text-to-3D synthesis results. However, the method has two inherent limitations: (a) extremely slow optimization of NeRF and (b) low-resolution image space supervision on NeRF, leading to low-quality 3D models with a long processing time. In this paper, we address these limitations by utilizing a two-stage optimization framework. First, we obtain a coarse model using a low-resolution diffusion prior and accelerate with a sparse 3D hash grid structure. Using the coarse representation as the initialization, we further optimize a textured 3D mesh model with an efficient differentiable renderer interacting with a high-resolution latent diffusion model. Our method, dubbed Magic3D, can create high quality 3D mesh models in 40 minutes, which is 2x faster than DreamFusion (reportedly taking 1.5 hours on average), while also achieving higher resolution. User studies show 61.7% raters to prefer our approach over DreamFusion. Together with the image-conditioned generation capabilities, we provide users with new ways to control 3D synthesis, opening up new avenues to various creative applications.

[2211.10440] Magic3D: High-Resolution Text-to-3D Content Creation Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Computer Vision and Pattern Recognition arXiv:2211.10440 (cs) [Submitted on 18 Nov 2022 ( v1 ), last revised 25 Mar 2023 (this version, v2)] Title: Magic3D: High-Resolution Text-to-3D Content Creation Authors: Chen-Hsuan Lin , Jun Gao , Luming Tang , Towaki Takikawa , Xiaohui Zeng , Xun Huang , Karsten Kreis , Sanja Fidler , Ming-Yu Liu , Tsung-Yi Lin View a PDF of the

related reading

• 2311.15980arxiv.org
• NeRF: Neural Radiance Fieldsmatthewtancik.com
• Neural radiance field - Wikipediaen.wikipedia.org
• ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenesecva.net
• RTFM: A Real-Time Frame Model | World Labsworldlabs.ai
• The Illustrated Stable Diffusion – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• ⭐️ Diffusion Modelsandrewkchan.dev
• 3D as code | World Labsworldlabs.ai
• 3DMV3dmv2023.github.io
• Introducing CM3leon, a more efficient, state-of-the-art generative model for text and imagesai.meta.com
• 2409.02908arxiv.org
• Point-NeRF: Point-based Neural Radiance Fieldsxharlie.github.io