FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization

Abstract

3DGS has demonstrated remarkable capabilities in real-time novel view synthesis tasks. However, it tends to experience over-reconstruction during Gaussian densification, particularly when high-variance image regions are being covered by a small number of large Gaussians typically resulting in blurring and artifacts in the rendered images.

We construct FreGS, an advanced method, to address over-reconstruction in the frequency domain.

In particular, FreGS had achieved the process of Gaussian densification through a coarse-to-fine approach by leveraging the low-to-high frequency components, which are readily obtainable using low-pass and high-pass filters in the Fourier space.

By reducing the difference between the frequency spectrum of rendered images and their corresponding ground truths, this approach achieves high-quality Gaussian convolution while mitigating excessive reconstruction in Gaussian splatting scenarios.

A series of experiments over diverse widely adopted benchmarks, including examples such as the Mip-NeRF 360-based system, Tanks and Temples-based framework, and Deep Image Blending technology, demonstrate that FreGS exhibits superior performance in novel view synthesis. The results consistently show that FreGS surpasses the state-of-the-art performance across various scenarios.

Figure

Figure 1

FreGS effectively reduces over-modeling of Gaussian densification, generating images that exhibit significantly reduced blur and artifacts compared to 3DGS.

Two sample images from Mip-NeRF360, labeled (a) and (b), demonstrate both a Rendered Image and a Gaussian Visualization of emphasized regions, alongside Spectra obtained from over-reconstructed areas within these images using 3DGS, complementing with corresponding data from Fre GS.

The Gaussian Visualization displays the composition of images through learned 3D Gaussians (where all Gaussians are rendered with full opacity). The Spectra are created using image Fourier transformation, resulting in a color transition from blue to green as the spectrum amplitude varies from low to high.

Figure 2

Overview of the proposed FreGS.

3\text{D} Gaussian distributions are initialized via Structure-from-Motion (SfM). Following the projection of 3\text{D} Gaussian distributions, we can yield 2\text{D} Gaussian distributions and then apply standard \alpha-mixing for rendering.

Frequency spectra

and

are generated by applying Fourier transform to rendered image

and ground truth

, respectively.

The process of Frequency regularization is accomplished through the control of variations in amplitude, ensuring consistent performance across different operational conditions.

and __**phase

**__ in Fourier space.

A novel annealing-based innovative methodology aims to implement incremental frequency regularization.

With low-pass filter

and dynamic high-pass filter

, low-to-high frequency components_ are being utilized in a progressive manner to execute coarse-and-fine Gaussian densification._

Considering that, the incremental frequency-based regularization acts as a complementary component to the pixel-wise reconstruction loss between

and

.

The red dotted line emphasizes the regularization process of Gaussian densification.

Figure 3

The average gradient magnitudes of pixels within the context of over-reconstructed areas and high-quality reconstructed zones are analyzed in the scene labeled as 'Bicycle'.

The curve with a circle (without frequency regularization (FR)) is comparable to the 3DGS approach, which entirely utilizes pixel-wise L1 loss in the spatial domain.

After terminating at the 15,000th iteration, which mirrors the approach in 3DGS, we present comparisons prior to reaching this threshold. It becomes evident that frequency regularization significantly increases pixel gradient magnitudes within over-reconstruction regions. Therefore, when contrasted with L¹ loss, frequency regularization demonstrates superior capability in identifying these over-reconstructed areas.

Figure 4

The comparison of different frequency regularizations.

The simple frequency regularization method directly utilizes amplitude and phase discrepancies while ignoring the distinction between low and high frequencies.

This method incorporates the frequency annealing technique to achieve ascending-order frequency regularization from coarse to fine Gaussian densification.

Notice that the proposed method of progressive frequency regularization enables more precise Gaussian densification and produces more advanced novel view synthesis.

Figure 5

Qualitative assessment of FreGS for novel view synthesis as compared to three state-of-the-art methods.

Considering that we aim to achieve a fair comparison while balancing the trade-off between synthesis quality and memory consumption, we train FreGS using the same amount of Gaussians as employed by 3DGS on these datasets.

‘GT’ represents the ground-truth images. The proposed method FreGS offers high-quality image rendering, featuring fewer artifacts and a greater level of detail.

Figure 6

Displaying the progression of Gaussian distribution densification combined with rendered images throughout the training procedure.

The FreGS method effectively enhances Gaussian-based densification in over-reconstructed areas, resulting in progressively improved rendered images that exhibit fewer artifacts and greater detail.

Both 3DGS and FreGS are visualized through a series of experiments at 7, 15, and the final 30 training episodes to evaluate performance improvements.

Upon reaching the 15,000th iteration, the count of Gaussian components stabilizes as no further changes occur in the system. This phenomenon is attributed to the termination of Gaussian densification.

Conclusion

This paper introduces FreGS, an innovative 3DGS, which employs progressive frequency regularization to enhance the 3DGS framework from the standpoint of frequency analysis.

Specifically, we develop afrequency annealing technique for progressive frequency regularization, which achieves gradual Gaussian densification from coarse to fine through the progressive utilization of low-frequency to high-frequency components, all of which are extractable using **low-pass and high-pass filters within the Fourier domain.

FreGS reduces the difference between the frequency spectrum of rendered images and their ground truth, thereby alleviating the over-reconstruction problem and achieving advanced Gaussian density enhancement.

Experiments over multiple widely adopted indoor and outdoor scenes show that FreGS achieves superior novel view synthesis and outperforms the SOTA consistently.