Combining the diffusion model with NeRF, Tsinghua Wensheng proposed a new 3D method to achieve SOTA

The AI ​​model that uses text to synthesize 3D graphics has a new SOTA!

Recently, the research group of Professor Liu Yongjin of Tsinghua University proposed a new method of creating 3D images based on the diffusion model.

Both the consistency between different perspectives and the matching with prompt words have been greatly improved compared to before.

Picture

Vincent 3D is a hot research content of 3D AIGC and has received widespread attention from academia and industry.

The new model proposed by Professor Liu Yongjin’s research team is called TICD (Text-Image Conditioned Diffusion), which has reached the SOTA level on the T3Bench data set.

Relevant papers have been released and the code will be open source soon.

The evaluation results have reached SOTA

In order to evaluate the effect of the TICD method, the research team first conducted qualitative experiments and compared some previous better methods.

The results show that the 3D graphics generated by the TICD method have better quality, clearer graphics, and a higher degree of matching with the prompt words.

Picture

To further evaluate the performance of these models, the team quantitatively tested TICD with these methods on the T3Bench dataset.

The results show that TICD achieved the best results in the three prompt sets of single object, single object with background, and multiple objects, proving that it has overall advantages in both generation quality and text alignment. .

Picture

In addition, in order to further evaluate the text alignment of these models, the research team also performed CLIP of the pictures rendered by the 3D object and the original prompt words Cosine similarity was tested, and the result was that TICD still performed best.

So, how does the TICD method achieve such an effect?

Incorporating multi-view consistency prior into NeRF supervision

Currently mainstream 3D text generation methods mostly use pre-trained 2D diffusion models and are optimized through Score Distillation Sampling (SDS) Neural Radiation Field (NeRF) to generate brand new 3D models.

However, the supervision provided by this pre-trained diffusion model is limited to the input text itself, and does not constrain the consistency between multiple views, and may cause problems such as poor generated geometric structures.

To introduce multi-view consistency in the prior of the diffusion model, some recent studies have fine-tuned 2D diffusion models by using multi-view data, but still lack fine-grained inter-view continuity.

To solve this challenge, the TICD method incorporates text-conditioned and image-conditioned multi-view images into the NeRF-optimized supervision signal, ensuring the alignment of 3D information with prompt words and 3D objects respectively. The strong consistency between viewing angles effectively improves the quality of generated 3D models.

Picture

In terms of workflow, TICD first samples several sets of orthogonal reference camera perspectives, uses NeRF to render the corresponding reference views, and then renders these The reference view uses a text-based conditional diffusion model to constrain the overall consistency of content and text.

On this basis, select several sets of reference camera perspectives, and render a view from an additional new perspective for each perspective. Then, the pose relationship between the two views and perspectives is used as a new condition, and an image-based conditional diffusion model is used to constrain the consistency of details between different perspectives.

Combining the supervision signals of the two diffusion models, TICD can update the parameters of the NeRF network and optimize iteratively until the final NeRF model is obtained and renders high-quality, geometrically clear and text-consistent 3D content.

In addition, the TICD method can effectively eliminate problems such as the disappearance of geometric information, excessive generation of incorrect geometric information, and color confusion that may occur when existing methods face specific text input.

Paper address: https://www.php.cn/link/8553adf92deaf5279bcc6f9813c8fdcc

The above is the detailed content of Combining the diffusion model with NeRF, Tsinghua Wensheng proposed a new 3D method to achieve SOTA. For more information, please follow other related articles on the PHP Chinese website!