Natural language is integrated into NeRF, and LERF, which generates 3D images with just a few words, is here.

NeRF (Neural Radiance Fields), also known as neural radiation fields, has quickly become one of the most popular research fields since it was proposed, and the results are amazing. However, the direct output of NeRF is only a colored density field, which provides little information to researchers. The lack of context is one of the problems that need to be faced. The effect is that it directly affects the construction of interactive interfaces with 3D scenes.

But natural language is different. Natural language interacts with 3D scenes very intuitively. We can use the kitchen scene in Figure 1 to explain that objects can be found in the kitchen by asking where the cutlery is, or asking where the tools used to stir are. However, completing this task requires not only the query capabilities of the model, but also the ability to incorporate semantics at multiple scales.

In this article, researchers from UC Berkeley proposed a novel method and named it LERF (Language Embedded Radiance Fields), which combines CLIP (Contrastive Language-Image Pre -training) are embedded into NeRF, making these types of 3D open language queries possible. LERF uses CLIP directly, without the need for fine-tuning through datasets such as COCO, or relying on masked region suggestions. LERF preserves the integrity of CLIP embeddings at multiple scales and is also able to handle a variety of linguistic queries, including visual attributes (e.g., yellow), abstract concepts (e.g., electric current), text, etc., as shown in Figure 1.

##Paper address: https://arxiv.org/pdf/2303.09553v1.pdf

Project homepage: https://www.lerf.io/

LERF can interactively provide languages ​​for real-time Prompt to extract 3D related diagrams. For example, on a table with a lamb and a water cup, enter the prompt lamb or water cup, and LERF can give the relevant 3D picture:

# #For complex bouquets, LERF can also pinpoint:

Different objects in the kitchen:

Method

This study constructed a new method LERF by jointly optimizing the language field with NeRF. LERF takes position and physical scale as input and outputs a single CLIP vector. During training, the fields are supervised using a multi-scale feature pyramid containing CLIP embeddings generated from image crops of the training views. This allows the CLIP encoder to capture image context at different scales, thereby associating the same 3D location with language embeddings at different scales. LERF can query the language field at any scale during testing to obtain a 3D correlation map.

#Since CLIP embeddings are extracted from multiple views at multiple scales, the correlation mapping of text queries obtained by LERF’s 3D CLIP embeddings is consistent with The one obtained through 2D CLIP embedding is more localized and 3D consistent, and can be queried directly in the 3D field without rendering multiple views.

LERF requires learning a language embedding field on a volume centered on a sample point. Specifically, the output of this field is the average CLIP embedding of all training views containing image crops of the specified volume. By reconstructing queries from points to volumes, LERF can effectively supervise dense fields from coarse crops of input images, which can be rendered in a pixel-aligned manner by conditioning on a given volumetric scale.

#LERF itself produces coherent results, but the resulting correlation map can sometimes be incomplete and contain some outliers, as shown in Figure 5 below.

To standardize the optimized language field, this study introduces self-supervised DINO through shared bottlenecks.

In terms of architecture, optimizing language embedding in 3D should not affect the density distribution in the underlying scene representation, so this study captures the inductive bias in LERF by training two independent networks. Settings (inductive bias): one for feature vectors (DINO, CLIP) and one for standard NeRF output (color, density).

Experiment

To demonstrate LERF’s ability to process real-world data, the study collected 13 scenes, including grocery stores, kitchens, bookstores, figurines, etc. . Figure 3 selects 5 representative scenarios to demonstrate LERF’s ability to process natural language.

##Figure 3

Figure 7 is 3D visual comparison of LERF and LSeg. In the eggs in the calibration bowl, LSeg is inferior to LERF:

Figure 8 shows that under limited segmentation data LSeg trained on the set lacks the ability to effectively represent natural language. Instead, it only performs well on common objects within the training set distribution, as shown in Figure 7.

However, the LERF method is not perfect yet. The following are failure cases. For example, when calibrating zucchini vegetables, other vegetables will appear:

The above is the detailed content of Natural language is integrated into NeRF, and LERF, which generates 3D images with just a few words, is here.. For more information, please follow other related articles on the PHP Chinese website!