AIxiv專欄是本站發布學術、技術內容的欄位。過去數年，本站AIxiv專欄接收通報了2,000多篇內容，涵蓋全球各大專院校與企業的頂尖實驗室，有效促進了學術交流與傳播。如果您有優秀的工作想要分享，歡迎投稿或聯絡報道。投稿信箱：liyazhou@jiqizhixin.com；zhaoyunfeng@jiqizhixin.com
本論文作者包括帝國理工學院碩士生楊潤一、北航二年級碩士生朱貞欣、北京理工大學二年級碩士生薑洲、北京理工大學四年級本科生葉柏均、中國科學院大學本科大三學生張逸飛、中國電信人工智慧研究院多媒體認知學習實驗室（EVOL Lab）負責人趙健、清華大學智慧產業研究院（AIR）助理教授趙昊等#。 最近，3D Gaussian Splatting (3DGS) 作為一種新穎的 3D 表示方式，因其快速的渲染速度和高渲染品質而受到關注。然而，這種方法也伴隨著高記憶體消耗，例如，一個訓練好的高斯場可能會使用超過三百萬個高斯基元和超過 700 MB 的記憶體。 近日，帝國學院、北航、北京理工大學、中國科學院大學、中國電信人工智慧研究院多媒體認知學習實驗室（EVOL Lab）、清華大學智能產業研究院（AIR）等機構的研究者聯合發表了一篇論文《SUNDAE: Spectrally Pruned Gaussian Fields with Neural Compensation》，我們認為這種高內存佔用是由於沒有考慮基元之間的關係。在論文中，我們提出了一種名為 SUNDAE 的記憶體高效的高斯場，採用頻譜修剪和神經補償。 
- 文章連結：https://arxiv.org/abs/2405.00676
- 專案首頁： https://runyiyang.github.io/projects/SUNDAE/
#一方面，我們基於高斯基元的空間資訊建立了一個圖，用於模擬它們之間的關係，並設計了一個基於圖訊號處理的降採樣模組來剪枝，同時保留所需訊號。另一方面，為了補償剪枝造成的質量下降，我們利用了一個輕量級神經網路來混合渲染特徵，有效地補償了質量下降，同時在其權重中捕獲基元之間的關係。 我們透過大量的結果展示了 SUNDAE 的表現。例如，在 Mip-NeRF360 資料集上，SUNDAE 可以在使用 104 MB 記憶體的情況下達到 26.80 PSNR 和 145 FPS，而標準的 3D Gaussian Splatting 演算法在使用 523 MB 記憶體的情況下達到 25.60 PSNRPS 和 160 FNRPS。
同時，自從開源後，SUNDAE 受到國際上廣泛的關注，受到了知名NeRF 社群MrNeRF，AI research 社群維護者Ahsen Khaliq、以及多位相關領域研究人員轉發關注。 
#3DGS 使用一組高斯基元來表示場景，由於這些基元在三維空間中的分佈不規則，我們提出了基於圖的方法來捕捉基元之間的關係，而不是使用網格這樣的常規結構。 具體來說，我們採用圖訊號處理理論來推導一個最優的取樣策略，該策略能夠基於圖訊號保留特定頻譜的資訊。透過控制頻譜頻寬，我們可以靈活地控制剪枝比例，建模高斯基元之間的關係。如圖 1 (c)，我們可以控制剪枝 90% 的高斯基元而不降低渲染品質。 
圖1: (a) 3DGS 7k 迭代次數的結果；(b) 3DGS 30k 迭代次數的結果，使用了更多的高斯基元來表示三維場景，因此質量更高、速度更慢、儲存空間更大；(c) 剪枝了90% 的高斯基元，在儲存空間上大幅減小，但是達到了相似的渲染效果。 我們使用高斯基元的中心來作為圖上的訊號輸入，將高斯基元之間的距離作為圖的邊，圖的鄰接矩陣可以表示為
其中
是高斯基元的中心點，
是一個閾值超參數，
是距離矩陣的變異數。也就是說，如果兩個高斯基元之間的距離比一個閾值小，那麼我們將其之間建立一條圖的邊。在建立好圖的鄰接矩陣之後，我們可以根據 Haar-like 濾波器對圖上的訊號進行處理，得到特定頻段的圖訊號。最後根據想要的頻段訊號進行剪枝，本文我們使用了帶阻濾波器，保留表示物體細節的高頻訊號和背景點的低頻訊號。 #經過頻譜剪枝後，渲染品質因為刪去了過多的高斯基元不可避免會下降，為了解決這個問題，我們採用了一個神經網路來補償這個品質損失，如圖2 所示。 我們從Gaussian Splatting 轉換到了Feature Splatting，引入一個輕量級卷積神經網路來輸出高斯基元映射到影像上的RGB 值，從而融合不同基元的資訊。這使得補償網路的權重在二維影像空間中間接地來捕捉基元之間的關係。 
圖2: 左邊顯示的原版3DGS，由於沒有捕捉到基元之間的關係，因此需要大量的儲存空間；中間展示了我們的頻譜剪枝策略，建模高斯基元之間的關係；右側顯示神經補償利用2D 特徵來改善渲染效果。 具體來說，我們不是像3DGS 那樣直接渲染RGB 影像，而是透過用於3D 高斯的可微分光柵化器來獲得一個特徵圖，此光柵化器將3D 高斯基元的特徵投影到二維特徵圖上。 然後，我們利用一個輕量級神經網路來建模基元之間的關係並補償頻譜剪枝後的質量下降。這個網路由一個具有 skip-connection 的四層全卷積 U-Net 組成，它聚合來自不同基元的資訊。使用平均池化進行下取樣，並使用雙線性內插法進行影像上取樣。此網路以光柵化的特徵圖為輸入，輸出 RGB 影像。 
Figure 3: (a) Pipeline: For a pre-trained 3D Gaussian field, a graph-based pruning strategy is used to downsample the Gaussian elements, and a convolutional neural network is used to compensate for the pruning. Damage caused by branches. (b) Graph-based pruning: Graphs based on spatial relationships between Gaussian primitives are used for pruning. By using band-stop filters, this process facilitates the extraction of detailed information from high-frequency components while capturing general features of the low-frequency portion, resulting in a comprehensive and efficient representation of the entire scene. 1.3 Continuous Pruning StrategyIn addition, we A continuous pruning strategy is also proposed to reduce peak storage. Unlike post-training pruning, which prunes primitives from a fully dense Gaussian field, continuous pruning involves regular shifts at predefined intervals throughout the training process. Except for a specific number or proportion of primitives. This approach aims to continuously control the maximum number of primitives when training 3D Gaussian fields, thereby reducing peak memory requirements during training and allowing training on GPU devices with low GPU memory. Experience shows that the advantages of lower peak memory come at the expense of weaker final memory footprint control. For example, if we prune 20% of the primitives every 2000 iterations, the final convergence state of the 3D Gaussian field may deviate from the expected 20% reduction. Additionally, this variation may vary across scenarios, adding complexity to the predictability and consistency of pruning effects. Therefore, we consider the continuous pruning strategy as an alternative when necessary. ##2.1 Quantitative resultsWe compared SUNDAE with the state-of-the-art 3DGS and NeRF algorithms. Compared to 3DGS, our model only takes up 10% of the memory to achieve similar results. effect, and can exceed the original 3DGS by using 30% or 50% of the memory. And it far exceeds other NeRF-related algorithms in terms of FPS.
This is because our model can better capture the relationship between Gaussian primitives and use fewer Gaussian primitives to efficiently represent the three-dimensional scene.
As you can see from the qualitative results, we Qualitative results of SUNDAE at 1% and 10% sampling rates are compared with 3DGS and InstantNGP.
Qualitative results show that SUNDAE is able to achieve similar new perspective synthesis quality while using only 10% or even 1% of the memory consumption. The graph successfully builds relationships between primitives, while the neural compensation head effectively maintains rendering quality. And as can be seen from the fourth and last row of Figure 5, spectrum pruning is able to remove floating objects close to the camera.
2.3.1 Band stop filter
The ratio of the band stop filter is represented by a parameter. Specifically, during the graph-based pruning process, we sample several primitives, including a certain proportion () of high-pass and the remaining (1-) low-pass.
The results show that this parameter has a significant impact on rendering quality, with a ratio of 50% providing the best results, while a disproportionate emphasis on low or high frequency signals will Resulting in a loss of quality, the effect is better because the 50% ratio retains balanced high-frequency details and low-frequency background.
2.3.2 Compensation Network
As shown in Figure 6 and Table 2, we demonstrate qualitatively and quantitatively Understand the importance of compensation network. As shown in Table 2, using neural compensation shows improved performance compared to not using it at all sampling rates. This is further supported by the visualization results presented in Figure 6 , demonstrating the module’s ability to compensate for the performance degradation caused by spectrum pruning. At the same time, it is also proved that the relationship between primitives is well captured.
As shown in Table 3, we tried different sizes of compensation networks, and increasing the network size does not necessarily improve rendering quality, which is consistent with the findings of ADOP, indicating a similar trend. We use 30MB of 4-tier UNet as the default setting for the best balance of quality and memory.
2.3.3 More sampling pointsAs shown in Table 1 above, retain 50% of the primitives Better than original 3DGS in rendering quality. We also additionally tested retaining 80% and retaining all primitives to examine how the sampling rate affects the final results, as shown in Table 4. The results show that retaining 80% of the primitives improves rendering quality, showing improvement in terms of LPIPS, but little visual improvement in PSNR and SSIM. Retaining all primitives (and training for more epochs) does not improve the quality further, which also shows the importance of modeling primitive relationships. Without effective relational modeling, more primitives make it difficult for the model to converge, and a large number of primitives has a negative impact on scene representation. Additionally, we aimed to balance rendering quality with storage efficiency; however, increasing storage to 620MB to preserve 80% of primitives only resulted in a slight quality improvement , thereby reducing storage efficiency. 
2.3.4 Continuous sampling strategyWe tested continuous sampling on the Bicycle and Counter scenarios in the MipNeRF360 dataset Sampling strategy, setting different pruning interval iterations and pruning rates. As shown in Table 5, Points is the number of primitives after training, and Ratio is the approximate ratio of the number of primitives after training to the original 3DGS. The results show that this strategy can reduce the peak memory, but it is difficult to control the final memory (reflected by Points and Ratio). Therefore, we validated our post-training pruning strategy but still provide the continuous pruning strategy as an alternative in our open source toolbox. 
2.3.5 Efficiency EvaluationAbout training time, CUDA memory, rendering frame rate and ROM storage See Table 6 for details. It is worth noting that the “Ours-50%” version achieved the best rendering quality within acceptable training time (1.41 hours), while achieving real-time rendering and significantly reducing CUDA memory usage and ROM storage during training. . 
In this work, we proposed a The novel spectral pruned Gaussian field SUNDAE with neural compensation introduces graph signal processing to model the relationship between Gaussian primitives and mixes the information of different primitives to compensate for the information loss caused by pruning. We use the spatial information between Gaussian primitives to construct a graph to model relationships, and prune based on the spectral information to remove redundant primitives. A lightweight neural network is used to compensate for the inevitable loss of rendering quality after pruning. Experimental results show that SUNDAE significantly reduces memory, improves efficiency and maintains high-fidelity rendering quality while maintaining the efficiency of 3DGS. 以上是告別3D高斯Splatting演算法，帶有神經補償的頻譜剪枝高斯場SUNDAE開源了的詳細內容。更多資訊請關注PHP中文網其他相關文章！
