The 3D version of SORA is here! DreamTech launches Direct3D, the world’s first native 3D-DiT large model

In May 2024, DreamTech officially announced its high-quality 3D generation large model Direct3D, and published the related academic paper Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer.

Link: https://arxiv.org/abs/2405.14832

This is the first publicly released By using 3D Diffusion Transformer (3D-DiT) to generate large 3D models of native 3D routes, the problem of high-quality 3D content generation that has long plagued the industry has been solved.

Adhere to the native 3D technology route and achieve breakthroughs

Previously, 3D The technical route usually adopted by AIGC is 2D-to-3D lifting, which is to obtain a 3D model by upgrading the dimension of the 2D image model. Representative solutions include the early Score Distillation Sampling (SDS) represented by DreamFusion proposed by Google, and Adobe's The proposed Instant3D is represented by Large Reconstruction Model (LRM). Although 3D data is gradually introduced into the model training process to improve quality, 2D dimensionality enhancement technology has inherent problems such as multiple heads and faces, cavities, and occlusions. Existing solutions are difficult to meet the requirements of commercial applications for general 3D generation.

At the beginning of last year, some people in the industry began to try the native 3D route, that is, to obtain 3D models directly without going through intermediate multi-view 2D pictures or multi-view iterative optimization. This technical route It can avoid the shortcomings of 2D dimensionality enhancement, showing the potential to obtain high-quality, non-deformed, non-deformed, commercially available 3D content. In principle, native 3D routes have significant advantages over 2D dimensionality enhancement methods. However, there have always been many challenges in model training and algorithm development. The most critical issues are:

1. Efficient 3D model representation: Images and videos can directly obtain latent features through 2D/2.5D matrix representation compression. In contrast, 3D data has complex topology and higher representation dimensions. How to efficiently compress 3D data and then analyze and learn the distribution of 3D data in 3D latent space is a problem that has always troubled industry personnel.

2. Efficient 3D training architecture: DiT architecture was first applied in the field of image generation and achieved great success, including Stable Diffusion 3 (SD3), Hunyuan-DiT All adopt the DiT architecture; in the field of video generation, OpenAI SORA uses the DiT architecture to successfully achieve video generation effects that far exceed Runway and Pika; in the field of 3D generation, limited by complex topology and three-dimensional representation methods, the original DiT architecture cannot directly Applied to 3D mesh generation.

3. High-quality large-scale 3D training data: The quality and scale of 3D training data directly determine the quality and generalization ability of the generated model. It is generally believed in the industry that at least Tens of millions of high-quality 3D training data are needed to meet the training requirements of large 3D models. However, 3D data is extremely scarce around the world. Although there are tens of millions of 3D training data sets such as ObjaverseXL, most of them are low-quality simple structures, and the available high-quality 3D data accounts for less than 5 %. How to obtain a sufficient amount of high-quality 3D data is a worldwide problem.

In response to the above core problems, DreamTech proposed the world's first native 3D-DiT large model Direct3D. Through extensive experimental verification, the 3D model generation quality of Direct3D significantly surpasses the current mainstream 2D ​​dimensionality method, which mainly benefits from the following three points:

1. D3D-VAE : Direct3D proposes a 3D VAE (Variational Auto-Encoder) similar to OpenAI SORA to extract latent features of 3D data, reducing the representation complexity of 3D data from the original N^3 to n^2 (n<< N) compact 3D latent space, and achieves nearly lossless recovery of the original 3D mesh through the decoder network. By using the 3D latent feature, Direct3D reduces the original computational and memory requirements for training 3D-DiT by more than two orders of magnitude, making large-scale 3D-DiT model training possible.

2. D3D-DiT: Direct3D adopts the DiT architecture and improves and optimizes the original DiT. It introduces semantic-level and pixel-level alignment modules for input images to achieve output The model is aligned to the height of any input image.

3. DreamTech 3D Data Engine: Direct3D uses a large amount of high-quality 3D data in training, most of which is produced by DreamTech's self-developed data synthesis engine become. The DreamTech synthesis engine has established fully automatic data processing processes such as data cleaning and annotation, and has accumulated and produced more than 20 million high-quality 3D data, completing the last piece of the puzzle for the implementation of native 3D algorithms. It is worth mentioning that OpenAI tried to use millions of 3D synthetic data in the training process of Shap-E and Point-E in 2023. Compared with OpenAI’s data synthesis solution, the 3D data synthesized by DreamTech is larger in scale, and Higher quality.

Adopt DiT architecture

3D field re-verifies Scaling Law

In terms of technical architecture, Direct3D uses Diffusion Transformer (DiT), which is similar to OpenAI SORA. The DiT architecture is currently the most advanced AIGC large model architecture. It combines the advantages of the two major architectures of Diffusion and Transformer to meet the requirements of scalability, that is, it provides the model with more data and more large model parameters. DiT can achieve or even exceed human generative quality. Current practical projects of DiT technology include Stable Diffusion 3 (Stablility AI, February 2024), Hunyuan-DiT (Tencent, May 2024) in the direction of image generation, and SORA (OpenAI, February 2024) in the direction of video generation. ), DreamTech's Direct3D is the world's first public DiT practice in the direction of 3D content generation.

DiT architecture complies with and has been proven multiple times by the Scaling Law.

Scaling Law has fully proven its effectiveness on large language models. As the number of parameters and training data increases, the intelligence of large models will be greatly improved; In the field of image generation, from SD1 parameter size 0.8B to SD3 8B, Dall-E 3 parameter size 12B, all demonstrate the effectiveness of Scaling Law; in the field of video generation, SORA compared to Runway, Pika, etc., it is speculated that Its technical implementation mainly involves replacing the model architecture with DiT, and increasing the number of model parameters and training data by an order of magnitude, demonstrating a generation effect that shocks the world, whether it is video resolution, video duration or video generation quality. Has been greatly improved.

The same is true in the 3D field. Direct3D-1B shows the industry the first feasible native 3D-DiT architecture, using a self-developed high-quality data synthesis engine to increase training data. By increasing the amount of model parameters and increasing the amount of model parameters, the generation results have been steadily improved. In the future, Direct3D (or its derivative architecture) will completely replace the existing LRM or SDS solutions in the field of 3D generation. Currently, the DreamTech team is steadily promoting the scale up of Direct3D, and plans to launch Direct3D-XL with 15B parameters before the end of the year. At the same time, it will increase the high-quality 3D data for training models by more than 5 times. 3D generation will usher in a milestone moment.

The quality of 3D content generation reaches commercial level

With the launch of Direct3D, 3D The field of generation has made great strides into the commercial era. Taking 3D printing as an example, models generated using technical solutions such as SDS and LRM will have the following problems:

1. The model geometry is distorted and prone to long heads and tails;

2. The model has many sharp burrs;

3. The surface is overly smooth and lacks details;

4. The mesh has a small number of patches. Fine structure cannot be guaranteed.

The existence of these problems has caused the models generated by various previous solutions to be unable to be printed normally on 3D printers, and require manual adjustments and repairs. Because Direct3D adopts the native 3D technology route and only uses 3D data in the training set, the quality of the 3D models it generates is closer to the original quality, and it perfectly solves core issues such as geometric structure, model accuracy, surface details, and number of mesh patches. The quality of the models generated by Direct3D has exceeded the upper limit of accuracy of home printers. Only commercial and industrial printers with higher specifications can fully restore the precision of the generated models.

Previously, technical solutions such as SDS and LRM were limited by the expression form of 3D model features. Generally, the number of generated model mesh patches was around 50,000-200,000, and it was difficult to increase it. However, in commercial use, 3D models The number of mesh patches often needs to reach more than 1-5 million. Direct3D proposes a more refined 3D feature expression paradigm, so that the number of generated model meshes has no upper limit and can reach and exceed 10 million, meeting the needs of various business scenarios.

As the amount of Direct3D model parameters and training data increases, 3D generation can be applied to more and more industries, including trillion-level games and animation industries. , it is expected that before the end of 2025, 3D generation will realize the replacement work of most games, animations, film and television modeling, and be put into large-scale use in various industries.

Direct3D Practice

Based on the Direct3D large model, DreamTech has launched two early adopter products , currently open for application testing (Click to read the original text, jump to: www.neural4d.com).

One is Animeit! for C-end users. Animeit! can convert any picture/text object input by the user into a high-quality 3D character image in a two-dimensional style. And 3D characters have skeletal nodes for action binding. On Animeit!, users can directly talk to and interact with personalized 3D AI partners.

Animeit! The two-dimensional characters generated are extremely precise, with clearly identifiable facial contour details, prominent hand details, and distinct fingertips. This is unprecedented in the past. A level of quality that cannot be achieved by the 3D generation technology route can be used for MMD production in the two-dimensional community.

Another product is a 3D content creation platform for creators. Users can use platforms like Midjourney to Text description: Get a high-quality 3D model within 1 minute, without waiting for a long refinement; users can also just upload a single picture, and wait a moment to get a high-quality and accurately restored 3D model.

About DreamTech

##DreamTech is deeply involved in the field of 3D AI technology and is committed to innovation The company's products and services enhance the experience of global AIGC creators and consumers. The company's vision is to use advanced AI technology to create a 4D space-time experience that is seamlessly connected to the real world and interactive in real time, and to simulate the complexity and diversity of the real world. Achieving general artificial intelligence (AGI).

#DreamTech brings together the world's top AI talents. Its founding team is composed of academicians of the British Academy and the Chinese Academy of Sciences, national-level young talents, and many high-level talents in Shenzhen. The core members of the company graduated from world-renowned universities such as Oxford University, the Chinese University of Hong Kong, and the Hong Kong University of Science and Technology, and have worked in industry-leading companies such as Apple, Tencent, and Baidu. The founding team members have successfully founded a number of companies that have become benchmarks in the 3D field. These companies were later acquired by industry giants such as Apple, Google, and Bosch.

The above is the detailed content of The 3D version of SORA is here! DreamTech launches Direct3D, the world’s first native 3D-DiT large model. For more information, please follow other related articles on the PHP Chinese website!