Make 3D editing as easy as PS, the new algorithm GaussianEditor can complete the addition, deletion and modification of 3D scenes in a few minutes

3D editing plays a vital role in fields such as games and virtual reality. However, previous 3D editing suffered from problems such as long time consumption and poor controllability, making it difficult to apply to actual scenes. Recently, Nanyang Technological University, Tsinghua University and SenseTime proposed a new 3D editing algorithm, GaussianEditor, which for the first time achieved controllable and diversified editing of 3D scenes in 2-7 minutes, completely surpassing previous 3D editing work.

In recent years, research in the field of 3D editing has generally focused on neural radiation fields (NeRF). This is because NeRF can not only perform 3D scene modeling with a high degree of fidelity, but its implicit characteristics greatly improve scalability, which has significant advantages over traditional point cloud and mesh methods. However, NeRF relies on high-dimensional multi-layer perceptron networks (MLP) to encode scene data, which also brings certain limitations. It makes it difficult to directly modify specific parts of a scene and increases the complexity of tasks such as image restoration and scene composition. This complexity not only affects the training process, but also limits its use in practical applications.

GaussianEditor In order to solve the above problems, we took a new approach and chose Gaussian sputtering as the Its 3D representation. Gaussian Splatting is a new type of 3D representation proposed half a year ago. This representation has surpassed NeRF in many 3D tasks such as 3D and 4D reconstruction. It has attracted widespread attention in the 3D field as soon as it was launched and is one of the biggest breakthroughs in the 3D field this year. one. Gaussian Splatting has excellent prospects and potential, and GaussianEditor is the first to implement editing of this 3D representation. The project is open source and provides a WebUI interface for easy learning and use.

• ##Paper address: https://arxiv.org/abs/2311.14521
• Home page address: https://buaacyw.github.io/gaussian-editor/

Although Gaussian Splatting has an efficient rendering algorithm, it is used as a display representation Editing presents quite a few challenges. A major problem is the lack of efficient methods to accurately identify editing targets, which is crucial for precise and controllable editing. Furthermore, it has been shown that there are significant challenges in optimizing Gaussian spraying (GS) using highly stochastic generative guidance, such as generative diffusion models such as Stable Diffusion. This may be because GS is directly affected by the randomness in the loss, unlike the implicit representation of neural network buffering. This direct exposure leads to unstable updates and the properties of Gaussian points change directly during training. In addition, each training step of GS may involve the updating of a large number of Gaussian points, and this process does not have a neural network-style buffering mechanism. These problems will cause the excessive fluidity of GS to hinder its convergence to as fine a result as the implicit representation during training.

In order to solve the above problems, The team first introduced Gaussian semantic tracking to achieve precise control of Gaussian Splatting (GS). Gaussian semantic tracking can always identify Gaussian points that need to be edited during the training process. This differs from traditional 3D editing methods, which often rely on static 2D or 3D masks. As the geometry and appearance of the 3D model changes during training, these masks gradually become ineffective. Gaussian semantic tracking achieves tracking throughout the training process by projecting 2D segmentation masks onto 3D Gaussian points and assigning semantic labels to each Gaussian point. As the Gaussian points change during training, these semantic labels enable tracking of specific target Gaussian points. Gaussian semantic tracking algorithm ensures that only target areas are modified, enabling precise and controlled editing.

The red area in the figure below is the tracked target area. The semantic tracking area will be dynamically updated with the training process to ensure its effectiveness.

In addition, in order to deal with the major challenge of Gaussian Sputtering (GS) being difficult to achieve fine results when it is highly randomly generated, GaussinEditor adopts a new GS representation: hierarchical Gaussian sputtering (Hierarchical Gaussian Splatting, HGS). In HGS, Gaussian points are organized into different generations based on their densification order during training. Gaussian points formed during earlier densification processes are considered older generations, and they are subject to tighter constraints with the aim of maintaining their original state and reducing their mobility. In contrast, Gaussian points formed in later stages are treated as younger generations subject to fewer or no constraints to improve their fitness. The design of HGS effectively regulates the mobility of GS by imposing restrictions on older generations while maintaining the flexibility of newer generations. This approach makes possible continuous optimization towards better results, simulating the buffering function in implicit representations implemented through neural networks

GaussianEditor proposed the addition and deletion algorithm of Gaussian sputtering representation on this basis. In terms of target deletion, the team developed a specialized local repair algorithm that effectively eliminates artifacts at the interface between the object and the scene. In terms of adding targets, GaussianEditor can add specified targets to specified areas based on a text prompt and 2D mask provided by the user. GaussianEditor first generates a single-view image of the object to be added with the help of the 2D Image Inpainting algorithm. This image is then converted into a 3D GS using the Image to 3D algorithm. Finally, the target is incorporated into the Gaussian scene.

In comparative experiments, GaussianEditor significantly surpassed previous work in terms of visual quality, quantitative indicators, controllability and generation speed

#The team also verified the effectiveness of their proposed Gaussian semantic tracking and hierarchical Gaussian representation through ablation experiments

GaussianEditor As an advanced 3D editing algorithm, the focus is on editing 3D scenes flexibly and quickly, and Editing of Gaussian sputtering is implemented for the first time.

The key features of the algorithm include:

1. Gaussian Semantic Tracking: It can continuously identify Gaussian points that need to be edited during the training process, ensuring that only the target area is edited.
2. Hierarchical Gaussian Splatting (HGS) : This is a new GS representation, through Gaussian formed in different training stages A hierarchy is established between the points to effectively manage the fluidity of the GS scene and simulate the buffering function of the neural network in the implicit representation.
3. 3D scene addition and deletion algorithm: GaussianEditor has specially developed and designed a 3D scene addition and deletion algorithm for GS, which can efficiently remove scenes from scenes. Remove or add specific objects.

The above is the detailed content of Make 3D editing as easy as PS, the new algorithm GaussianEditor can complete the addition, deletion and modification of 3D scenes in a few minutes. For more information, please follow other related articles on the PHP Chinese website!