search
HomeTechnology peripheralsAIOne line of text to achieve 3D face-changing! UC Berkeley proposes 'Chat-NeRF' to complete blockbuster-level rendering in just one sentence

Thanks to the development of neural 3D reconstruction technology, capturing feature representations of real-world 3D scenes has never been easier.

However, there has never been a simple and effective solution for 3D scene editing above this.

Recently, researchers from UC Berkeley proposed a method of editing NeRF scenes using text instructions - Instruct-NeRF2NeRF, based on the previous work InstructPix2Pix.

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

##Paper address: https://arxiv.org/abs/2303.12789

Using Instruct-NeRF2NeRF, we can edit large-scale real-world scenes with just one sentence, and make it more realistic and targeted than previous work.

For example, if you want him to have a beard, a tuft of beard will appear on his face!

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

Or just change your head and become Einstein in seconds.

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

#In addition, since the model can continuously update the data set with new edited images, the reconstruction of the scene will gradually improve.

NeRF InstructPix2Pix = Instruct-NeRF2NeRF

Specifically, humans are given an input image, and written instructions that tell the model what to do, which the model then follows. These instructions are used to edit images.

The implementation steps are as follows:

    Render an image from the scene in the training perspective.
  1. Use the InstructPix2Pix model to edit this image based on global text instructions.
  2. Replace the original images in the training dataset with the edited images.
  3. The NeRF model continues training as usual.

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence


Implementation

Compared with traditional 3D editing, NeRF2NeRF is a new 3D scene editing method. Its biggest highlight is the use of "iterative data set update" technology.

Although editing is performed on a 3D scene, a 2D rather than a 3D diffusion model is used in the paper to extract form and appearance priors because the data used to train the 3D generative model is very limited. .

This 2D diffusion model is the InstructPix2Pix developed not long ago by the research team - a 2D image editing model based on command text. When you input image and text commands, it can output editing image after.

However, this 2D model will cause uneven changes in different angles of the scene. Therefore, "iterative data set update" came into being. This technology alternately modifies NeRF's "input image data". Set" and update the underlying 3D representation.

This means that the text-guided diffusion model (InstructPix2Pix) will generate new image variations according to the instructions and use these new images as input for NeRF model training. Therefore, the reconstructed 3D scene will be based on new text-guided editing.

In the initial iterations, InstructPix2Pix often fails to perform consistent edits across different views, however, during NeRF re-rendering and updating, they will converge to a globally consistent Scenes.

In summary, the NeRF2NeRF method improves the editing efficiency of 3D scenes by iteratively updating image content and integrating these updated contents into the 3D scene, while also maintaining Scene coherence and realism.

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

It can be said that this work of the UC Berkeley research team is an extended version of the previous InstructPix2Pix. By combining NeRF with InstructPix2Pix and working with "iterative data set update", a Key editing can still play with 3D scenes!

There are still limitations, but the flaws are not hidden

However, since Instruct-NeRF2NeRF is based on the previous InstructPix2Pix, it inherits many limitations of the latter, such as the inability to carry out large-scale space operations.

Additionally, like DreamFusion, Instruct-NeRF2NeRF can only use the diffusion model on one view at a time, so you may encounter similar artifact issues.

The following figure shows two types of failure cases:

(1) Pix2Pix cannot perform editing in 2D, so NeRF2NeRF cannot perform editing in 3D It also failed;

(2) Pix2Pix can complete editing in 2D, but there is a big inconsistency in 3D, so NeRF2NeRF also failed.

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

Another example is the "panda" below, which not only looks very fierce (the prototype statue is very fierce) , and the fur color is somewhat weird, and the eyes are obviously "out of shape" when moving in the screen.

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

Since ChatGPT, Diffusion, and NeRFs have been pulled into the spotlight, this article can be said to give full play to the advantages of the three, from "AI Sentence "Word drawing" has advanced to "AI one-sentence editing of 3D scenes".

Although the method has some limitations, it still has its flaws and provides a simple and feasible solution for 3D feature editing, which is expected to become a milestone in the development of NeRF.

Editing 3D scenes in one sentence

Finally, let’s take a look at the effects released by the author.

It is not difficult to see that this one-click PS 3D scene editing artifact is more in line with expectations in terms of command understanding ability and image realism. In the future, it may become a popular choice among academics and The "new favorite" among netizens has created Chat-NeRFs after ChatGPT.

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

Even if you change the environmental background, seasonal characteristics, and weather of the image at will, The new images given are also completely consistent with the logic of reality.

Original picture:

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

##Autumn:

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence


Snow day:

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

## Desert:

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

storm:

One line of text to achieve 3D face-changing! UC Berkeley proposes Chat-NeRF to complete blockbuster-level rendering in just one sentence

Reference: https://www .php.cn/link/ebeb300882677f350ea818c8f333f5b9

The above is the detailed content of One line of text to achieve 3D face-changing! UC Berkeley proposes 'Chat-NeRF' to complete blockbuster-level rendering in just one sentence. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:51CTO.COM. If there is any infringement, please contact admin@php.cn delete
How to Run LLM Locally Using LM Studio? - Analytics VidhyaHow to Run LLM Locally Using LM Studio? - Analytics VidhyaApr 19, 2025 am 11:38 AM

Running large language models at home with ease: LM Studio User Guide In recent years, advances in software and hardware have made it possible to run large language models (LLMs) on personal computers. LM Studio is an excellent tool to make this process easy and convenient. This article will dive into how to run LLM locally using LM Studio, covering key steps, potential challenges, and the benefits of having LLM locally. Whether you are a tech enthusiast or are curious about the latest AI technologies, this guide will provide valuable insights and practical tips. Let's get started! Overview Understand the basic requirements for running LLM locally. Set up LM Studi on your computer

Guy Peri Helps Flavor McCormick's Future Through Data TransformationGuy Peri Helps Flavor McCormick's Future Through Data TransformationApr 19, 2025 am 11:35 AM

Guy Peri is McCormick’s Chief Information and Digital Officer. Though only seven months into his role, Peri is rapidly advancing a comprehensive transformation of the company’s digital capabilities. His career-long focus on data and analytics informs

What is the Chain of Emotion in Prompt Engineering? - Analytics VidhyaWhat is the Chain of Emotion in Prompt Engineering? - Analytics VidhyaApr 19, 2025 am 11:33 AM

Introduction Artificial intelligence (AI) is evolving to understand not just words, but also emotions, responding with a human touch. This sophisticated interaction is crucial in the rapidly advancing field of AI and natural language processing. Th

12 Best AI Tools for Data Science Workflow - Analytics Vidhya12 Best AI Tools for Data Science Workflow - Analytics VidhyaApr 19, 2025 am 11:31 AM

Introduction In today's data-centric world, leveraging advanced AI technologies is crucial for businesses seeking a competitive edge and enhanced efficiency. A range of powerful tools empowers data scientists, analysts, and developers to build, depl

AV Byte: OpenAI's GPT-4o Mini and Other AI InnovationsAV Byte: OpenAI's GPT-4o Mini and Other AI InnovationsApr 19, 2025 am 11:30 AM

This week's AI landscape exploded with groundbreaking releases from industry giants like OpenAI, Mistral AI, NVIDIA, DeepSeek, and Hugging Face. These new models promise increased power, affordability, and accessibility, fueled by advancements in tr

Perplexity's Android App Is Infested With Security Flaws, Report FindsPerplexity's Android App Is Infested With Security Flaws, Report FindsApr 19, 2025 am 11:24 AM

But the company’s Android app, which offers not only search capabilities but also acts as an AI assistant, is riddled with a host of security issues that could expose its users to data theft, account takeovers and impersonation attacks from malicious

Everyone's Getting Better At Using AI: Thoughts On Vibe CodingEveryone's Getting Better At Using AI: Thoughts On Vibe CodingApr 19, 2025 am 11:17 AM

You can look at what’s happening in conferences and at trade shows. You can ask engineers what they’re doing, or consult with a CEO. Everywhere you look, things are changing at breakneck speed. Engineers, and Non-Engineers What’s the difference be

Rocket Launch Simulation and Analysis using RocketPy - Analytics VidhyaRocket Launch Simulation and Analysis using RocketPy - Analytics VidhyaApr 19, 2025 am 11:12 AM

Simulate Rocket Launches with RocketPy: A Comprehensive Guide This article guides you through simulating high-power rocket launches using RocketPy, a powerful Python library. We'll cover everything from defining rocket components to analyzing simula

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Tools

SecLists

SecLists

SecLists is the ultimate security tester's companion. It is a collection of various types of lists that are frequently used during security assessments, all in one place. SecLists helps make security testing more efficient and productive by conveniently providing all the lists a security tester might need. List types include usernames, passwords, URLs, fuzzing payloads, sensitive data patterns, web shells, and more. The tester can simply pull this repository onto a new test machine and he will have access to every type of list he needs.

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!

PhpStorm Mac version

PhpStorm Mac version

The latest (2018.2.1) professional PHP integrated development tool