search
HomeTechnology peripheralsAIThe 3D version of SORA is here! DreamTech launches Direct3D, the world's first native 3D-DiT large model

3D 版 SORA 来了!DreamTech 推出全球首个原生 3D-DiT 大模型 Direct3D

In May 2024, DreamTech officially announced its high-quality 3D generation large model Direct3D, and published the related academic paper Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer.

3D 版 SORA 来了!DreamTech 推出全球首个原生 3D-DiT 大模型 Direct3D

Link: https://arxiv.org/abs/2405.14832

This is the first publicly released By using 3D Diffusion Transformer (3D-DiT) to generate large 3D models of native 3D routes, the problem of high-quality 3D content generation that has long plagued the industry has been solved.

3D 版 SORA 来了!DreamTech 推出全球首个原生 3D-DiT 大模型 Direct3D

Adhere to the native 3D technology route and achieve breakthroughs

Previously, 3D The technical route usually adopted by AIGC is 2D-to-3D lifting, which is to obtain a 3D model by upgrading the dimension of the 2D image model. Representative solutions include the early Score Distillation Sampling (SDS) represented by DreamFusion proposed by Google, and Adobe's The proposed Instant3D is represented by Large Reconstruction Model (LRM). Although 3D data is gradually introduced into the model training process to improve quality, 2D dimensionality enhancement technology has inherent problems such as multiple heads and faces, cavities, and occlusions. Existing solutions are difficult to meet the requirements of commercial applications for general 3D generation.

3D 版 SORA 来了!DreamTech 推出全球首个原生 3D-DiT 大模型 Direct3D

At the beginning of last year, some people in the industry began to try the native 3D route, that is, to obtain 3D models directly without going through intermediate multi-view 2D pictures or multi-view iterative optimization. This technical route It can avoid the shortcomings of 2D dimensionality enhancement, showing the potential to obtain high-quality, non-deformed, non-deformed, commercially available 3D content. In principle, native 3D routes have significant advantages over 2D dimensionality enhancement methods. However, there have always been many challenges in model training and algorithm development. The most critical issues are:
  1. Efficient 3D model representation: Images and videos can directly obtain latent features through 2D/2.5D matrix representation compression. In contrast, 3D data has complex topology and higher representation dimensions. How to efficiently compress 3D data and then analyze and learn the distribution of 3D data in 3D latent space is a problem that has always troubled industry personnel.

  2. Efficient 3D training architecture: DiT architecture was first applied in the field of image generation and achieved great success, including Stable Diffusion 3 (SD3), Hunyuan-DiT All adopt the DiT architecture; in the field of video generation, OpenAI SORA uses the DiT architecture to successfully achieve video generation effects that far exceed Runway and Pika; in the field of 3D generation, limited by complex topology and three-dimensional representation methods, the original DiT architecture cannot directly Applied to 3D mesh generation.

  3. High-quality large-scale 3D training data: The quality and scale of 3D training data directly determine the quality and generalization ability of the generated model. It is generally believed in the industry that at least Tens of millions of high-quality 3D training data are needed to meet the training requirements of large 3D models. However, 3D data is extremely scarce around the world. Although there are tens of millions of 3D training data sets such as ObjaverseXL, most of them are low-quality simple structures, and the available high-quality 3D data accounts for less than 5 %. How to obtain a sufficient amount of high-quality 3D data is a worldwide problem.

3D 版 SORA 来了!DreamTech 推出全球首个原生 3D-DiT 大模型 Direct3D

In response to the above core problems, DreamTech proposed the world's first native 3D-DiT large model Direct3D. Through extensive experimental verification, the 3D model generation quality of Direct3D significantly surpasses the current mainstream 2D ​​dimensionality method, which mainly benefits from the following three points:

  1. D3D-VAE : Direct3D proposes a 3D VAE (Variational Auto-Encoder) similar to OpenAI SORA to extract latent features of 3D data, reducing the representation complexity of 3D data from the original N^3 to n^2 (n

  2. D3D-DiT: Direct3D adopts the DiT architecture and improves and optimizes the original DiT. It introduces semantic-level and pixel-level alignment modules for input images to achieve output The model is aligned to the height of any input image.

  3. DreamTech 3D Data Engine: Direct3D uses a large amount of high-quality 3D data in training, most of which is produced by DreamTech's self-developed data synthesis engine become. The DreamTech synthesis engine has established fully automatic data processing processes such as data cleaning and annotation, and has accumulated and produced more than 20 million high-quality 3D data, completing the last piece of the puzzle for the implementation of native 3D algorithms. It is worth mentioning that OpenAI tried to use millions of 3D synthetic data in the training process of Shap-E and Point-E in 2023. Compared with OpenAI’s data synthesis solution, the 3D data synthesized by DreamTech is larger in scale, and Higher quality.

Adopt DiT architecture
3D field re-verifies Scaling Law

In terms of technical architecture, Direct3D uses Diffusion Transformer (DiT), which is similar to OpenAI SORA. The DiT architecture is currently the most advanced AIGC large model architecture. It combines the advantages of the two major architectures of Diffusion and Transformer to meet the requirements of scalability, that is, it provides the model with more data and more large model parameters. DiT can achieve or even exceed human generative quality. Current practical projects of DiT technology include Stable Diffusion 3 (Stablility AI, February 2024), Hunyuan-DiT (Tencent, May 2024) in the direction of image generation, and SORA (OpenAI, February 2024) in the direction of video generation. ), DreamTech's Direct3D is the world's first public DiT practice in the direction of 3D content generation.

DiT architecture complies with and has been proven multiple times by the Scaling Law.

3D 版 SORA 来了!DreamTech 推出全球首个原生 3D-DiT 大模型 Direct3D

Scaling Law has fully proven its effectiveness on large language models. As the number of parameters and training data increases, the intelligence of large models will be greatly improved; In the field of image generation, from SD1 parameter size 0.8B to SD3 8B, Dall-E 3 parameter size 12B, all demonstrate the effectiveness of Scaling Law; in the field of video generation, SORA compared to Runway, Pika, etc., it is speculated that Its technical implementation mainly involves replacing the model architecture with DiT, and increasing the number of model parameters and training data by an order of magnitude, demonstrating a generation effect that shocks the world, whether it is video resolution, video duration or video generation quality. Has been greatly improved.

The same is true in the 3D field. Direct3D-1B shows the industry the first feasible native 3D-DiT architecture, using a self-developed high-quality data synthesis engine to increase training data. By increasing the amount of model parameters and increasing the amount of model parameters, the generation results have been steadily improved. In the future, Direct3D (or its derivative architecture) will completely replace the existing LRM or SDS solutions in the field of 3D generation. Currently, the DreamTech team is steadily promoting the scale up of Direct3D, and plans to launch Direct3D-XL with 15B parameters before the end of the year. At the same time, it will increase the high-quality 3D data for training models by more than 5 times. 3D generation will usher in a milestone moment.

The quality of 3D content generation reaches commercial level

With the launch of Direct3D, 3D The field of generation has made great strides into the commercial era. Taking 3D printing as an example, models generated using technical solutions such as SDS and LRM will have the following problems:
  1. The model geometry is distorted and prone to long heads and tails;

  2. The model has many sharp burrs;

  3. The surface is overly smooth and lacks details;

  4. The mesh has a small number of patches. Fine structure cannot be guaranteed.

The existence of these problems has caused the models generated by various previous solutions to be unable to be printed normally on 3D printers, and require manual adjustments and repairs. Because Direct3D adopts the native 3D technology route and only uses 3D data in the training set, the quality of the 3D models it generates is closer to the original quality, and it perfectly solves core issues such as geometric structure, model accuracy, surface details, and number of mesh patches. The quality of the models generated by Direct3D has exceeded the upper limit of accuracy of home printers. Only commercial and industrial printers with higher specifications can fully restore the precision of the generated models.

3D 版 SORA 来了!DreamTech 推出全球首个原生 3D-DiT 大模型 Direct3D

Previously, technical solutions such as SDS and LRM were limited by the expression form of 3D model features. Generally, the number of generated model mesh patches was around 50,000-200,000, and it was difficult to increase it. However, in commercial use, 3D models The number of mesh patches often needs to reach more than 1-5 million. Direct3D proposes a more refined 3D feature expression paradigm, so that the number of generated model meshes has no upper limit and can reach and exceed 10 million, meeting the needs of various business scenarios.

As the amount of Direct3D model parameters and training data increases, 3D generation can be applied to more and more industries, including trillion-level games and animation industries. , it is expected that before the end of 2025, 3D generation will realize the replacement work of most games, animations, film and television modeling, and be put into large-scale use in various industries.

Direct3D Practice

Based on the Direct3D large model, DreamTech has launched two early adopter products , currently open for application testing (Click to read the original text, jump to: www.neural4d.com).

One is Animeit! for C-end users. Animeit! can convert any picture/text object input by the user into a high-quality 3D character image in a two-dimensional style. And 3D characters have skeletal nodes for action binding. On Animeit!, users can directly talk to and interact with personalized 3D AI partners.

Animeit! The two-dimensional characters generated are extremely precise, with clearly identifiable facial contour details, prominent hand details, and distinct fingertips. This is unprecedented in the past. A level of quality that cannot be achieved by the 3D generation technology route can be used for MMD production in the two-dimensional community. 3D 版 SORA 来了!DreamTech 推出全球首个原生 3D-DiT 大模型 Direct3D3D 版 SORA 来了!DreamTech 推出全球首个原生 3D-DiT 大模型 Direct3D
Another product is a 3D content creation platform for creators. Users can use platforms like Midjourney to Text description: Get a high-quality 3D model within 1 minute, without waiting for a long refinement; users can also just upload a single picture, and wait a moment to get a high-quality and accurately restored 3D model.

About DreamTech

##DreamTech is deeply involved in the field of 3D AI technology and is committed to innovation The company's products and services enhance the experience of global AIGC creators and consumers. The company's vision is to use advanced AI technology to create a 4D space-time experience that is seamlessly connected to the real world and interactive in real time, and to simulate the complexity and diversity of the real world. Achieving general artificial intelligence (AGI).

#DreamTech brings together the world's top AI talents. Its founding team is composed of academicians of the British Academy and the Chinese Academy of Sciences, national-level young talents, and many high-level talents in Shenzhen. The core members of the company graduated from world-renowned universities such as Oxford University, the Chinese University of Hong Kong, and the Hong Kong University of Science and Technology, and have worked in industry-leading companies such as Apple, Tencent, and Baidu. The founding team members have successfully founded a number of companies that have become benchmarks in the 3D field. These companies were later acquired by industry giants such as Apple, Google, and Bosch.

The above is the detailed content of The 3D version of SORA is here! DreamTech launches Direct3D, the world's first native 3D-DiT large model. For more information, please follow other related articles on the PHP Chinese website!

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
DSA如何弯道超车NVIDIA GPU?DSA如何弯道超车NVIDIA GPU?Sep 20, 2023 pm 06:09 PM

你可能听过以下犀利的观点:1.跟着NVIDIA的技术路线,可能永远也追不上NVIDIA的脚步。2.DSA或许有机会追赶上NVIDIA,但目前的状况是DSA濒临消亡,看不到任何希望另一方面,我们都知道现在大模型正处于风口位置,业界很多人想做大模型芯片,也有很多人想投大模型芯片。但是,大模型芯片的设计关键在哪,大带宽大内存的重要性好像大家都知道,但做出来的芯片跟NVIDIA相比,又有何不同?带着问题,本文尝试给大家一点启发。纯粹以观点为主的文章往往显得形式主义,我们可以通过一个架构的例子来说明Sam

阿里云通义千问14B模型开源!性能超越Llama2等同等尺寸模型阿里云通义千问14B模型开源!性能超越Llama2等同等尺寸模型Sep 25, 2023 pm 10:25 PM

2021年9月25日,阿里云发布了开源项目通义千问140亿参数模型Qwen-14B以及其对话模型Qwen-14B-Chat,并且可以免费商用。Qwen-14B在多个权威评测中表现出色,超过了同等规模的模型,甚至有些指标接近Llama2-70B。此前,阿里云还开源了70亿参数模型Qwen-7B,仅一个多月的时间下载量就突破了100万,成为开源社区的热门项目Qwen-14B是一款支持多种语言的高性能开源模型,相比同类模型使用了更多的高质量数据,整体训练数据超过3万亿Token,使得模型具备更强大的推

ICCV 2023揭晓:ControlNet、SAM等热门论文斩获奖项ICCV 2023揭晓:ControlNet、SAM等热门论文斩获奖项Oct 04, 2023 pm 09:37 PM

在法国巴黎举行了国际计算机视觉大会ICCV(InternationalConferenceonComputerVision)本周开幕作为全球计算机视觉领域顶级的学术会议,ICCV每两年召开一次。ICCV的热度一直以来都与CVPR不相上下,屡创新高在今天的开幕式上,ICCV官方公布了今年的论文数据:本届ICCV共有8068篇投稿,其中有2160篇被接收,录用率为26.8%,略高于上一届ICCV2021的录用率25.9%在论文主题方面,官方也公布了相关数据:多视角和传感器的3D技术热度最高在今天的开

复旦大学团队发布中文智慧法律系统DISC-LawLLM,构建司法评测基准,开源30万微调数据复旦大学团队发布中文智慧法律系统DISC-LawLLM,构建司法评测基准,开源30万微调数据Sep 29, 2023 pm 01:17 PM

随着智慧司法的兴起,智能化方法驱动的智能法律系统有望惠及不同群体。例如,为法律专业人员减轻文书工作,为普通民众提供法律咨询服务,为法学学生提供学习和考试辅导。由于法律知识的独特性和司法任务的多样性,此前的智慧司法研究方面主要着眼于为特定任务设计自动化算法,难以满足对司法领域提供支撑性服务的需求,离应用落地有不小的距离。而大型语言模型(LLMs)在不同的传统任务上展示出强大的能力,为智能法律系统的进一步发展带来希望。近日,复旦大学数据智能与社会计算实验室(FudanDISC)发布大语言模型驱动的中

百度文心一言全面向全社会开放,率先迈出重要一步百度文心一言全面向全社会开放,率先迈出重要一步Aug 31, 2023 pm 01:33 PM

8月31日,文心一言首次向全社会全面开放。用户可以在应用商店下载“文心一言APP”或登录“文心一言官网”(https://yiyan.baidu.com)进行体验据报道,百度计划推出一系列经过全新重构的AI原生应用,以便让用户充分体验生成式AI的理解、生成、逻辑和记忆等四大核心能力今年3月16日,文心一言开启邀测。作为全球大厂中首个发布的生成式AI产品,文心一言的基础模型文心大模型早在2019年就在国内率先发布,近期升级的文心大模型3.5也持续在十余个国内外权威测评中位居第一。李彦宏表示,当文心

AI技术在蚂蚁集团保险业务中的应用:革新保险服务,带来全新体验AI技术在蚂蚁集团保险业务中的应用:革新保险服务,带来全新体验Sep 20, 2023 pm 10:45 PM

保险行业对于社会民生和国民经济的重要性不言而喻。作为风险管理工具,保险为人民群众提供保障和福利,推动经济的稳定和可持续发展。在新的时代背景下,保险行业面临着新的机遇和挑战,需要不断创新和转型,以适应社会需求的变化和经济结构的调整近年来,中国的保险科技蓬勃发展。通过创新的商业模式和先进的技术手段,积极推动保险行业实现数字化和智能化转型。保险科技的目标是提升保险服务的便利性、个性化和智能化水平,以前所未有的速度改变传统保险业的面貌。这一发展趋势为保险行业注入了新的活力,使保险产品更贴近人民群众的实际

致敬TempleOS,有开发者创建了启动Llama 2的操作系统,网友:8G内存老电脑就能跑致敬TempleOS,有开发者创建了启动Llama 2的操作系统,网友:8G内存老电脑就能跑Oct 07, 2023 pm 10:09 PM

不得不说,Llama2的「二创」项目越来越硬核、有趣了。自Meta发布开源大模型Llama2以来,围绕着该模型的「二创」项目便多了起来。此前7月,特斯拉前AI总监、重回OpenAI的AndrejKarpathy利用周末时间,做了一个关于Llama2的有趣项目llama2.c,让用户在PyTorch中训练一个babyLlama2模型,然后使用近500行纯C、无任何依赖性的文件进行推理。今天,在Karpathyllama2.c项目的基础上,又有开发者创建了一个启动Llama2的演示操作系统,以及一个

快手黑科技“子弹时间”赋能亚运转播,打造智慧观赛新体验快手黑科技“子弹时间”赋能亚运转播,打造智慧观赛新体验Oct 11, 2023 am 11:21 AM

杭州第19届亚运会不仅是国际顶级体育盛会,更是一场精彩绝伦的中国科技盛宴。本届亚运会中,快手StreamLake与杭州电信深度合作,联合打造智慧观赛新体验,在击剑赛事的转播中,全面应用了快手StreamLake六自由度技术,其中“子弹时间”也是首次应用于击剑项目国际顶级赛事。中国电信杭州分公司智能亚运专班组长芮杰表示,依托快手StreamLake自研的4K3D虚拟运镜视频技术和中国电信5G/全光网,通过赛场内部署的4K专业摄像机阵列实时采集的高清竞赛视频,

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks agoBy尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
1 months agoBy尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
4 weeks agoBy尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)