RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS

Abstract

Recent advances in view synthesis and real-time rendering have achieved outstanding results with impressive speed. While Radiance Field-based methods excel in complex scenarios like wild captures and large-scale setups, they are often burdened by excessive computational demands tied to volumetric rendering. On the other hand, Gaussian Splatting-based approaches utilize rasterization to achieve natural real-time performance yet struggle with fragile optimization strategies that perform poorly in more intricate environments.

Within this study, we introduce RadSplat, a computationally efficient solution for high-fidelity scene reconstruction. Our principal contributions include

Introduce radiance fields as a foundational model with supervision signals to refine point-based scene representations, resulting in enhanced fidelity and more reliable refinement.

Firstly, we introduce a new pruning method aimed at lowering the total number of points without compromising on quality. This approach results in more compact scene representations that enable faster inference.

Finally, we introduce an innovative runtime filtering technique that significantly enhances the rendering process and enables scaling to larger housescale scenes. We find that our method achieves SOTA performance in synthesizing complex captures at 900+ frames per second.

Figure

Figure 1

RadSplat achieves high-fidelity real-time view generation for large-scale indoor portions with a frame rate exceeding 900 fps.

(bottom) In contrast to 3DGS, our method allows for robust synthesis of complex captures while rendering 3,000× faster than the SOTA inoffline view synthesis, Zip-NeRF.

Figure 2

Overview.

From the perspective of the posed input images within a highly intricate real-world environment, we utilize a high-quality neural radiance field to incorporate GLO embeddings.

.

We use the radiance field before initializing and supervising our point-based 3DGS model that we utilize a novel pruning technique to enhance for more compact, high-quality scenes.

By implementing viewpoint-based visibility filtering, we significantly enhance the efficiency of test-time rendering processes.

Figure 3

Qualitative Comparison.

We show results on the mip-NeRF 360 dataset and Zip-NeRF dataset.

Unlike Zip-NeRF, our approach demonstrates superior performance by more effectively capturing not only high-frequency texture characteristics but also intricate geometric features. Specifically, it excels at depicting fine textures like tablecloths and carpets seen in kitchens and living rooms, as well as the detailed geometry of leaves, flowers, and bicycle wheels observed in urban settings such as Berlin.

It is compared to 3DGS, achieving clearer results such as grass below a bicycle bench and shiny surfaces in London, while more stable reconstructions are reached, such as the color shift observed in the kitchen.

Figure 4

Robustness.

(a) In cases of complicated scenes involving lighting variations, the 3DGS method exhibits poor performance outcomes.

When the exposure handling modules are installed, the results experience improvement, though they still encounter issues like floating artifacts and become excessively smooth.

Our radiance field-based approach instead delivers exceptional performance despite capturing intricate and complex scenes.

Figure 5

Ablation Study.

改写说明

Without supervision based on NeRFs, floating artifacts may arise when the views undergo changes in lighting or exposure settings (b). And without pruning, the scene point numbers are substantially higher despite no corresponding quality gains (c).

Figure 6

Pruning and Optimization Behavior.

(a)

(b) We analyze the initial optimization trajectory between 3DGS and our default model (pruning threshold of 0.01) within the "Bicycle" setting. Our findings reveal a more pronounced improvement in SSIM metrics, demonstrating that our default model achieves comparable quality within fewer than 8,000 optimization steps.

Conclusion and Limitations.

We optimize both the radiance field and the implicit surface model. In doing so, we achieve state-of-the-art performance at an impressive frame rate of over 900 FPS, yet our training duration of approximately two hours surpasses that of single-representation-based approaches, which typically require only half to one hour for training.

Further, although our method achieves outstanding performance on the MipNeRF360 dataset, there remains a slight discrepancy in terms of PSNR and LPIPS metrics when comparing to ZipNeRF in the context of large-scale ZipNeRF scenes. Our future research efforts will focus on exploring how to enhance both training efficiency and output quality for large residential and corporate-scale scenes.

We introduced RadSplat, a method incorporating the unique advantages of radiance fields and Gaussian Splatting, enabling highly robust real-time rendering of complex scenes at an impressive 900+ frames per second.

We showed that by utilizing radiance fields to serve as both a prior constraint and a supervision signal, we achieved enhanced outcomes in the optimization of point-based 3DGS representations.

The novel pruning approach enables more concise and efficient scenes while substantially reducing the number of points and enhancing the visual fidelity.

Finally, our innovative approach introduces runtime filtering techniques, which significantly enhance rendering speed without compromising performance. Our method demonstrates leading-edge performance on standard benchmarks, achieving up to 3,000 times faster rendering compared to prior approaches.