


One line of text to achieve 3D face-changing! UC Berkeley proposes 'Chat-NeRF' to complete blockbuster-level rendering in just one sentence
Thanks to the development of neural 3D reconstruction technology, capturing feature representations of real-world 3D scenes has never been easier.
However, there has never been a simple and effective solution for 3D scene editing above this.
Recently, researchers from UC Berkeley proposed a method of editing NeRF scenes using text instructions - Instruct-NeRF2NeRF, based on the previous work InstructPix2Pix.
##Paper address: https://arxiv.org/abs/2303.12789
Using Instruct-NeRF2NeRF, we can edit large-scale real-world scenes with just one sentence, and make it more realistic and targeted than previous work.
For example, if you want him to have a beard, a tuft of beard will appear on his face!
Or just change your head and become Einstein in seconds.
#In addition, since the model can continuously update the data set with new edited images, the reconstruction of the scene will gradually improve.
NeRF InstructPix2Pix = Instruct-NeRF2NeRFSpecifically, humans are given an input image, and written instructions that tell the model what to do, which the model then follows. These instructions are used to edit images.
The implementation steps are as follows:
- Render an image from the scene in the training perspective.
- Use the InstructPix2Pix model to edit this image based on global text instructions.
- Replace the original images in the training dataset with the edited images.
- The NeRF model continues training as usual.
Implementation
Compared with traditional 3D editing, NeRF2NeRF is a new 3D scene editing method. Its biggest highlight is the use of "iterative data set update" technology.
Although editing is performed on a 3D scene, a 2D rather than a 3D diffusion model is used in the paper to extract form and appearance priors because the data used to train the 3D generative model is very limited. .
This 2D diffusion model is the InstructPix2Pix developed not long ago by the research team - a 2D image editing model based on command text. When you input image and text commands, it can output editing image after.
However, this 2D model will cause uneven changes in different angles of the scene. Therefore, "iterative data set update" came into being. This technology alternately modifies NeRF's "input image data". Set" and update the underlying 3D representation.
This means that the text-guided diffusion model (InstructPix2Pix) will generate new image variations according to the instructions and use these new images as input for NeRF model training. Therefore, the reconstructed 3D scene will be based on new text-guided editing.
In the initial iterations, InstructPix2Pix often fails to perform consistent edits across different views, however, during NeRF re-rendering and updating, they will converge to a globally consistent Scenes.
In summary, the NeRF2NeRF method improves the editing efficiency of 3D scenes by iteratively updating image content and integrating these updated contents into the 3D scene, while also maintaining Scene coherence and realism.
It can be said that this work of the UC Berkeley research team is an extended version of the previous InstructPix2Pix. By combining NeRF with InstructPix2Pix and working with "iterative data set update", a Key editing can still play with 3D scenes!
There are still limitations, but the flaws are not hidden
However, since Instruct-NeRF2NeRF is based on the previous InstructPix2Pix, it inherits many limitations of the latter, such as the inability to carry out large-scale space operations.
Additionally, like DreamFusion, Instruct-NeRF2NeRF can only use the diffusion model on one view at a time, so you may encounter similar artifact issues.
The following figure shows two types of failure cases:
(1) Pix2Pix cannot perform editing in 2D, so NeRF2NeRF cannot perform editing in 3D It also failed;
(2) Pix2Pix can complete editing in 2D, but there is a big inconsistency in 3D, so NeRF2NeRF also failed.
Another example is the "panda" below, which not only looks very fierce (the prototype statue is very fierce) , and the fur color is somewhat weird, and the eyes are obviously "out of shape" when moving in the screen.
Since ChatGPT, Diffusion, and NeRFs have been pulled into the spotlight, this article can be said to give full play to the advantages of the three, from "AI Sentence "Word drawing" has advanced to "AI one-sentence editing of 3D scenes".
Although the method has some limitations, it still has its flaws and provides a simple and feasible solution for 3D feature editing, which is expected to become a milestone in the development of NeRF.
Editing 3D scenes in one sentence
Finally, let’s take a look at the effects released by the author.
It is not difficult to see that this one-click PS 3D scene editing artifact is more in line with expectations in terms of command understanding ability and image realism. In the future, it may become a popular choice among academics and The "new favorite" among netizens has created Chat-NeRFs after ChatGPT.
Even if you change the environmental background, seasonal characteristics, and weather of the image at will, The new images given are also completely consistent with the logic of reality.
Original picture:
##Autumn:
Snow day:
Reference: https://www .php.cn/link/ebeb300882677f350ea818c8f333f5b9
The above is the detailed content of One line of text to achieve 3D face-changing! UC Berkeley proposes 'Chat-NeRF' to complete blockbuster-level rendering in just one sentence. For more information, please follow other related articles on the PHP Chinese website!

Running large language models at home with ease: LM Studio User Guide In recent years, advances in software and hardware have made it possible to run large language models (LLMs) on personal computers. LM Studio is an excellent tool to make this process easy and convenient. This article will dive into how to run LLM locally using LM Studio, covering key steps, potential challenges, and the benefits of having LLM locally. Whether you are a tech enthusiast or are curious about the latest AI technologies, this guide will provide valuable insights and practical tips. Let's get started! Overview Understand the basic requirements for running LLM locally. Set up LM Studi on your computer

Guy Peri is McCormick’s Chief Information and Digital Officer. Though only seven months into his role, Peri is rapidly advancing a comprehensive transformation of the company’s digital capabilities. His career-long focus on data and analytics informs

Introduction Artificial intelligence (AI) is evolving to understand not just words, but also emotions, responding with a human touch. This sophisticated interaction is crucial in the rapidly advancing field of AI and natural language processing. Th

Introduction In today's data-centric world, leveraging advanced AI technologies is crucial for businesses seeking a competitive edge and enhanced efficiency. A range of powerful tools empowers data scientists, analysts, and developers to build, depl

This week's AI landscape exploded with groundbreaking releases from industry giants like OpenAI, Mistral AI, NVIDIA, DeepSeek, and Hugging Face. These new models promise increased power, affordability, and accessibility, fueled by advancements in tr

But the company’s Android app, which offers not only search capabilities but also acts as an AI assistant, is riddled with a host of security issues that could expose its users to data theft, account takeovers and impersonation attacks from malicious

You can look at what’s happening in conferences and at trade shows. You can ask engineers what they’re doing, or consult with a CEO. Everywhere you look, things are changing at breakneck speed. Engineers, and Non-Engineers What’s the difference be

Simulate Rocket Launches with RocketPy: A Comprehensive Guide This article guides you through simulating high-power rocket launches using RocketPy, a powerful Python library. We'll cover everything from defining rocket components to analyzing simula


Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

AI Hentai Generator
Generate AI Hentai for free.

Hot Article

Hot Tools

SecLists
SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

EditPlus Chinese cracked version
Small size, syntax highlighting, does not support code prompt function

Zend Studio 13.0.1
Powerful PHP integrated development environment

SublimeText3 English version
Recommended: Win version, supports code prompts!

PhpStorm Mac version
The latest (2018.2.1) professional PHP integrated development tool