David is resurrected! NVIDIA recreates "Neural Langelo", 3D reconstructs muscle texture visible to the naked eye

Today, NVIDIA recreated Michelangelo’s “Neuralangelo” from the 16th century.

Look, Neuralangelo has "reproduced" a 3D version of the famous statue of David, with marble details and textures that are lifelike.

You know, the statue of David collected at the Academy of Fine Arts in Florence is only 3.96 meters tall, including the base. There are 5.5 meters.

It can even reconstruct the internal and external structure of a building, with roof tiles, glass panes, and various details reproduced one by one.

All of this is the magic of "Neuralangelo".

A new AI model proposed by researchers from NVIDIA and Johns Hopkins University uses neural networks to reconstruct 3D objects.

The latest research has been accepted by CVPR 2023.

Paper address: https://research.nvidia.com/labs/dir/neuralangelo/paper.pdf

In particular, Neuralangelo can reconstruct "high-fidelity large-scale scenes" from mobile phone videos and videos taken by drones.

Isn’t that the future, where you can easily turn videos of a city or even outer space into an immersive world, and then put it into a game to experience it?

Netizens exclaimed that NVIDIA has hacked the "Matrix" world!

Some people even claim that Apple’s XR technology, coupled with Neuralangelo, can create “new worlds” .

Effect Demonstration

Nvidia’s headquarters

##Old truck

##Statue of IgnatiusReconstructed 3D scene

Previous AI models often struggled to accurately capture repetitive texture patterns, uniform colors, and strong color changes when reconstructing 3D scenes.

To this end, the team proposed a new method - Neuralangelo - that combines the representation capabilities of multi-resolution 3D hash grids with neural surface rendering.

Last year, Nvidia researchers created a new tool, 3D MoMa, which makes it easy to turn photos into 3D objects.

NeuralAngelo builds on this concept, allowing the import of larger, more detailed spaces and objects. What makes it special is that it can accurately capture repeating texture patterns, homogeneous colors and strong color changes.

By using "real-time neurographic primitives", the core of NVIDIA's Instant NeRF technology, Neuralangelo can capture finer details.

The team’s approach relies on 2 key elements:

(1) Numerical gradients for computing higher-order derivatives as smoothing operations;

(2) Perform coarse-to-fine optimization on hash grids that control different levels of detail.

Even without auxiliary depth, Neuralangelo effectively recovers dense 3D surface structures from multi-view images with a fidelity that significantly exceeds previous methods, enabling the extraction of data from RGB videos. Capture and reconstruct detailed large-scale scenes.

Building NeuralAngelo

The NeuralAngelo model is built on multi-resolution hash encoding and SDF-based volume rendering.

Step 1: Use numerical gradients to calculate higher-order derivatives

It is possible to optimize beyond local cells by using numerical gradients with steps that match the spatial resolution of the hash grid. Compared with analytic gradient, numerical gradient plays a smoothing operation on SDF.

Step 2: Gradually refine the level of detail

By gradually reducing the step size of the numerical gradient and enabling a higher resolution hash grid, the optimization effect can better restore large areas of smoothness Surfaces and fine geometric details. This learning process can gradually improve the level of detail.

Step Three: Optimization

NeuralAngelo uses three optimization objectives:

##RGB synthesis loss

: RGB reconstruction loss between input image and synthetic image.

Eikonal loss

: Regularize the underlying SDF so that its surface normal is Unit regularity.

Curvature loss

: Regularize the underlying SDF so that the average curvature is not arbitrary get bigger.

"Neurolangelo" is built, so how does it work?

##It can be said that Neuralangelo restored the entire process of Michelangelo's portrayal of David:

· First, the model will select several frames from the 2D video of the object/scene taken from different angles, and thus "see" its depth, size and shape. The process is similar to how a sculptural artist would initially compose a composition from multiple angles.

· The model then creates a rough 3D representation of the scene, just as the artist begins chiseling the shape of the subject.

· Finally, the model is refined to render for greater clarity of detail, just as an artist meticulously embellishes the texture to mimic the texture of a fabric or human form.

In a qualitative comparison of DPU benchmarks, Neuralangelo produces more accurate and higher-fidelity surfaces.

The following are the quantitative results of Neuralangelo in the DTU dataset. The model achieved good reconstruction accuracy and image synthesis quality.

#In the qualitative comparison of different coarse-to-fine optimization schemes, when using analytical gradients AG and AG P, there are artifacts on the rough surface of the object .

When using digital gradient (NG), a better rough surface can be reconstructed and details are smoothed.

And NVIDIA’s solution (NG P) can produce smooth surfaces and fine details.

The final result is a 3D object or large object that can be used in virtual reality applications, digital twins or robotics development. scale scenario.

Nvidia said that Neuralangelo’s ability to convert the textures of complex materials, including the roughness of roof tiles and the smoothness of marble, from 2D videos into 3D objects significantly exceeds previous efforts. method.

Ming-Yu Liu, senior director of Nvidia Research and author of the paper, gave his thoughts on the significance of this research:

"Neuralangelo provides ’s 3D reconstruction capabilities will be a huge benefit to creators, helping them recreate the real world within the digital world. This tool will eventually allow developers to import detailed objects - whether they are small statues, or large buildings - into video games or industrial digital twins in a virtual environment.

Creative professionals can import these 3D objects into design applications for further editing for use in art, video game development, robotics, and Used in industrial digital twins and other fields.

Author introduction

Zhaoshuo Li

Li Zhaoshuo is currently a doctoral student in computer science at Johns Hopkins University. His supervisors are Professor Mathias Unberath and Professor Russell H Taylor.

He has a strong interest in computer vision, computer graphics, and deep learning. His research focuses on reproducing motion and structure from images.

In addition, he also has many hobbies, including being a videographer, a mental health promoter, a pet dog lover, a surfer, a skydiver, a snowboarder...

Chen-Hsuan Lin

##Chen-Hsuan Lin is a research scientist at NVIDIA Research, working in computer vision, computer graphics, and artificial intelligence.

He received his PhD in Robotics from Carnegie Mellon University and was awarded an NVIDIA Graduate Fellowship. Previously, he also interned at Facebook AI Research and Adobe Research.

Lin is very interested in solving problems of 3D reconstruction, view synthesis and 3D content production. Its research aims to give artificial intelligence systems human-level 3D perception and imagination capabilities by learning from Internet-scale visual data, moving towards true 3D spatial intelligence.

Hot comments from netizens

NVIDIA scientist Jim Fan said,

In order to let you understand the development of artificial intelligence in 3D modeling Speed: The field went from left (original NeRF reconstructed mesh) to right (Nvidia's Neuralangelo) in 3 years.

Transporting reality into high-fidelity simulations is no longer a dream.

The new Neuralangelo model is a beast and Nvidia has decided to retire our, R.I.P. photogrammetry software.

It’s like the “Michelangelo” of the digital world.

Some netizens also expressed their interest in knowing how much it costs to use?

We can use drones in factories and then send video to this model, make a digital twin, and use it to optimize our processes.

Regarding the significance of this technology, netizens believe that its impact on the gaming industry will be huge.

The above is the detailed content of David is resurrected! NVIDIA recreates "Neural Langelo", 3D reconstructs muscle texture visible to the naked eye. For more information, please follow other related articles on the PHP Chinese website!