3D for all users? Magic3D from Nvidia can create 3D models from text.

Nvidia researchers introduced Magic3D on Friday, an AI system that can create 3D models from text descriptions. Magic3D builds a 3D mesh model with colored texture in around 40 minutes after receiving a prompt like "A blue poison-dart frog sitting on a water lily." The generated model can be altered and utilized in video games or CGI artwork. Nvidia frames Magic3D as a response to DreamFusion, a text-to-3D model that Google researchers unveiled in September, in its academic article. Magic3D employs a two-stage technique to take a coarse model created in low resolution and optimize it to higher resolution, much how DreamFusion uses a text-to-image model to build a 2D image that is subsequently optimized into volumetric NeRF (Neural radiance field) data. The authors of the research claim that the resulting Magic3D approach can create 3D objects twice as quickly as DreamFusion.

3D meshes can also be edited using prompts in Magic3D. A base prompt and a low-resolution 3D model can be used to modify the model by changing the wording. The creators of Magic3D also show how to transform a 3D model into a 2D image that has the appearance of a 2D image, like a cubist painting, and how to maintain the same subject across multiple generations (a practice known as coherence). Nvidia's scholarly study did not include any Magic3D code.

Today's diffusion models, which use neural networks to create new material after intensive training on a body of data, feel like a natural step to be able to create 3D from text. The development of capable text-to-image models like DALL-E and Stable Diffusion as well as basic text-to-video generators from Google and Meta has just occurred in 2022. The aforementioned text-to-3D model DreamFusion was also introduced by Google two months ago, and since then, people have developed comparable methods to use as an open source model based on Stable Diffusion.

The researchers who developed Magic3D believe that technology will make it possible for everyone to create 3D models without the requirement for specialized training. Once developed, the resulting technology would hasten the creation of video games (and virtual reality) and perhaps even find use in special effects for movies and television shows. "We think with Magic3D, we can democratize 3D synthesis and open up everyone's creativity in 3D content production," the authors conclude near the end of their essay.