An efficient and generalizable Gaussian reconstruction framework that can quickly reason with only 3 views and complete optimization in 45 seconds.

The AIxiv column is a column where this site publishes academic and technical content. In the past few years, the AIxiv column of this site has received more than 2,000 reports, covering top laboratories from major universities and companies around the world, effectively promoting academic exchanges and dissemination. If you have excellent work that you want to share, please feel free to contribute or contact us for reporting. Submission email: liyazhou@jiqizhixin.com; zhaoyunfeng@jiqizhixin.com

3D reconstruction and new view synthesis technology are widely used in the fields of virtual reality and augmented reality. NeRF has achieved remarkable success in view synthesis by implicitly encoding scenes as ray scenes. However, its practicality is greatly limited by the fact that NeRF relies on the time-consuming point-by-point querying of dense collections for rendering. To solve this problem, some generalizable NeRF methods have emerged, aiming to reconstruct scenes from multiple views in a network feedforward manner. However, NeRF-based methods are speed-limited since they require querying a dense collection of points on rays for rendering. Recently, 3D Gaussian Splatting (3D-GS) uses anisotropic 3D Gaussians to display scenes and achieves real-time high-quality rendering through a differential rasterizer.

However, 3D-GS also relies on the optimization of each scene, which takes dozens of minutes per scene. In order to solve this problem, some generalized Gaussian reconstruction work has appeared subsequently, trying to generalize 3D-GS to unseen scenes. However, the training and rendering efficiency of these methods need to be improved and are mainly limited to the reconstruction of objects or human bodies.

Based on this, researchers from Huazhong University of Science and Technology, Nanyang Technological University, Greater Bay Area University and Shanghai Artificial Intelligence Laboratory jointly proposed an efficient and generalizable Gaussian reconstruction model called MVSGaussian , for new view synthesis of unseen general scenes. This model works by splitting the input image into multiple views and using a Gaussian process to estimate depth and texture information, and then uses a multi-view stereo matching algorithm to fuse the views and generate high-quality reconstruction results. This method achieves a good balance between reconstruction quality and computational efficiency, providing a new solution for future visual synthesis tasks

• Paper name: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo

• Paper address: https://arxiv.org/abs/2405.12218

• Project homepage: https://mvsgaussian.github.io/

• Open source code: https://github.com/TQTQliu/MVSGaussian

• Demo video: https://youtu.be/4TxMQ9RnHMA

This model is able to learn a 3D Gaussian representation of a scene from sparse multi-view images. By combining the advantages of multi-view stereo (MVS) display format geometric reasoning and Gaussian deep shot real-time rendering, MVSGaussian performs well in generalized reasoning and can achieve the best view rendering quality at the fastest speed. In addition, MVSGaussian also has significant advantages in scene-by-scene optimization, completing high-quality real-time rendering in just 45 seconds (about 1/10 of 3D-GS).

图 1 Whether it is generalized reasoning or optimizing the scene, MVSGAUSSIAN has shown obvious advantages in view quality, rendering speed and optimization time .

Figure 2 Comparison of the changes in rendering view quality with optimization time (number of iterations). Since the generalizable model provides good initialization, MVSGaussian can achieve high-quality view synthesis with shorter optimization time (fewer iterations).

Basic Principles

To design an efficient and generalizable Gaussian sputtering framework, we face the following key challenges:

1) Unlike NeRF, which uses implicit representation, 3D-GS explicitly uses millions of 3D Gaussian spheres to express the scene. When applying pretrained 3D-GS to unseen scenes, the parameters of the 3D Gaussian sphere, such as position and color, differ significantly. Designing a general representation to adapt to 3D-GS is a non-trivial task. ###

2) The generalizable NeRF method achieves impressive view synthesis effects through volume rendering. However, the generalization ability of Gaussian sputtering has not been fully explored. During the sputtering process, each Gaussian sphere contributes to multiple pixels in a certain area of ​​the image, and the color of each pixel is accumulated from the contributions of multiple Gaussian spheres. The color correspondence between Gaussian spheres and pixels is a more complex many-to-many relationship, which poses a challenge to the generalization ability of the model.

3) The generalizable NeRF method shows that further fine-tuning for specific scenarios can significantly improve the quality of the synthesized views, but this requires a lot of time-consuming optimization. Although 3D-GS is faster than NeRF, it still takes longer. Therefore, designing a method for rapid scene-by-scenario optimization based on generalizable models is a very promising research direction.

In response to the above challenges, we have given our solutions.

1) Since the position distribution of the Gaussian sphere corresponding to each scene is different, we use multi-view stereo (MVS) to explicitly model the geometry of the scene and infer the depth. Next, we encode features for the 3D points corresponding to the estimated depth to build a pixel-aligned Gaussian representation.

2) Based on the encoded features, we can decode them into Gaussian parameters through MLP to render the view using sputtering technology. However, we found that this approach has limited generalization ability. Our insight is that the sputtering modality introduces a complex many-to-many relationship in terms of color contribution, that is, between Gaussian spheres and pixels, which poses a challenge to generalization. Therefore, we propose a simple and effective depth-aware volume rendering method to enhance generalization ability, that is, using a single sampling point volume rendering method. The final rendered view is obtained by averaging the views rendered by the sputtering technique and the volume rendering technique.

3) The pre-trained generalizable model can generate a large number of 3D Gaussians from multiple perspectives, and these Gaussian point clouds can be used as initialization for subsequent scene-by-scene optimization. However, due to the inherent limitations of the MVS method, the depth predicted by the generalizable model may not be completely accurate, resulting in noise in the generated Gaussian point cloud. Directly stitching these Gaussian point clouds together will produce a lot of noise. Additionally, a large number of points will slow down subsequent optimization and rendering. An intuitive solution is to downsample the stitched point cloud. However, while reducing noise, it also reduces the number of valid points. Our insight is that a good aggregation strategy should reduce noise points and retain valid points as much as possible while ensuring that the total number of points is not too large. To this end, we introduce an aggregation strategy based on multi-view geometric consistency. Specifically, we follow the principle that the predicted depth of the same 3D point under different viewing angles should be consistent, and filter out noise points by calculating the reprojection error of Gaussian depths from different viewing angles.

^{Figure 3 Generalizable Gaussian sputtering framework. Feature pyramid network (FPN) is first used to extract features from the input view, these features are warped to the target perspective, a cost volume is constructed, and then depth is generated through 3D CNNs regularization. Next, for depth-corresponding 3D points, we build pixel-aligned Gaussian representations by aggregating multi-view and spatial information encoding features. These features are then decoded into Gaussian parameters and volume rendering parameters, which render two views, and the final result is the average of the two views.}

^{Figure 4 Consistent aggregation. Using a generalizable model to generate depth maps and Gaussian point clouds, we first perform a multi-view geometric consistency check on the depth map to obtain a mask for filtering unreliable points. Subsequently, the filtered point clouds are spliced ​​into one point cloud as an initialization for scene-by-scene optimization.}

Result comparison

This paper is evaluated on the widely used DTU, Real Forward-facing, NeRF Synthetic and Tanks and Temples datasets , reporting metrics such as PSNR, SSIM, LPIPS and FPS. In terms of generalization inference (Tables 1 and 2), MVSGaussian demonstrates superior performance, achieving better performance with the fastest speed and minimal memory overhead. In terms of scene-by-scene optimization (Table 3), MVSGaussian is able to achieve the best view synthesis effect in the shortest optimization time (about 1/10 of 3D-GS) and maintains a real-time rendering speed comparable to 3D-GS. Qualitative view and video comparisons also demonstrate MVSGaussian's ability to synthesize high-quality views with more scene detail and fewer artifacts. See the project homepage for more video results.

^{Table 1 DTU Test sets of generalized quantitative results.}

#                                                                                                                                                                                                           Table 2 Quantitative results of generalization on the Real Forward-facing, NeRF Synthetic and Tanks and Temples datasets.

#                                   Table 3 Quantitative results after scenario-by-scenario optimization.

## 图 Figure 5 The result comparison of generalized reasoning.

#                                                       Figure 6 Video comparison of generalization results

Figure 7 Optimized results comparison after the scene.

# This Figure 8 Video comparison after optimization of scenes.

Conclusion

In this paper, we proposed MVSGaussian, a novel generalizable Gaussian sputtering method for Reconstruct scenes from multiple views. Specifically, we leverage MVS to reason about the geometry and build a pixel-aligned Gaussian representation. Furthermore, we propose a hybrid Gaussian rendering method that combines efficient depth-aware volume rendering to enhance generalization capabilities. In addition to directly generalizing inference, our model can be quickly fine-tuned for specific scenarios. To achieve fast optimization, we introduce a multi-view geometry consistent aggregation strategy to provide high-quality initialization. Compared to generalizable NeRF, which typically requires tens of minutes of fine-tuning and seconds to render each image, MVSGaussian enables real-time rendering with higher synthesis quality.

In addition, compared with 3D-GS, MVSGaussian achieves better view synthesis effects while reducing training computational costs. Extensive experiments verify that MVSGaussian reaches the state-of-the-art in terms of generalization performance, real-time rendering speed, and fast scene-by-scene optimization. However, since MVSGaussian relies on multi-view stereo (MVS) for depth estimation, it inherits the limitations of MVS, such as reduced depth accuracy in areas with weak textures or specular reflections, resulting in degraded view quality.

The above is the detailed content of An efficient and generalizable Gaussian reconstruction framework that can quickly reason with only 3 views and complete optimization in 45 seconds.. For more information, please follow other related articles on the PHP Chinese website!