Multi-Scale 3D Gaussian Splatting for Anti-Aliased Rendering

Abstract

3D Gaussians have recently emerged as a highly efficient representation for 3D reconstruction and rendering. Despite its high rendering quality and speed at high resolutions, they both deteriorate drastically when rendered at lower resolutions or from far away camera position.

During low resolution or far away rendering, the pixel size of the image can fall below the Nyquist frequency compared to the screen size of each splatted 3D Gaussian and leads to aliasing effect.The rendering is also drasticallyslowed down by the sequential alpha blending of more splatted Gaussians per pixel.

在低分辨率或远距离渲染时，图像的像素尺寸相对于每个泼溅的3D高斯屏幕尺寸可以下降到奈奎斯特频率（ Nyquist frequency是为防止信号混叠需要定义最小采样频率 ）以下，并导致混叠效应。

To address these issues, we propose a multi-scale 3D Gaussian splatting algorithm, which maintains Gaussiansat different scales to represent the same scene.

Higher-resolution images are rendered with more small Gaussians, and lower-resolution images are rendered withfewer larger Gaussians.

With similar training time, our algorithm can achieve 13%-66% PSNR and 160%-2400% rendering speed improvement at 4×-128× scale rendering on Mip-NeRF360 dataset compared to the single scale 3D Gaussian splatting.

Figure

Figure 1

The rendering quality and speed of the original 3DGS deteriorate severely at low resolutions or from distant cameras due to aliasing. Conversely, our multi-scale 3D Gaussians representation utilizes selective rendering to achieve faster (160%−2400% at 128× resolution) and more accurate rendering at lower resolutions.

Figure 2

Overall pipeline of our algorithm.

At the early stage of training (left), small Gaussians below certain size threshold in each voxel are aggregated , enlarged and inserted into the scene at different resolution scale.

During rendering (right), the multi-scale Gaussians of the appropriate “pixel coverage ” at the current render resolution are selected for rendering. If the rendering resolution scale equals to the scale of the Gaussians, the expected “pixel coverage ” range of the Gaussians are updated accordingly.

The “pixel coverage ” of a Gaussian reflects the size of the Gaussian when splatted onto the screen space compared to the pixel size at the current rendering resolution.

Figure 3

Pixel coverage of a 3D Gaussian is its horizontal or vertical size, whichever is smaller measured by the level set.

Figure 4

Missing parts caused by naive small Gaussian filtering at different resolution scales.

Figure 5

Large Gaussians are created by aggregating the small Gaussians in each voxel below the pixel coverage threshold , and then enlarged by the pixel coverage multiplier.

Figure 6

Based on the rendering resolution, the current pixel coverage of a Gaussian relative to its minimum and maximum pixel coverages determines whether it is selected for rendering.

Figure 7

Qualitative Comparison on 360 dataset for different resolution scales.

Figure 8

Qualitative Comparison on Tank and Temples dataset for different resolution scales.

Figure 9

Qualitative Comparison on Deep Blending dataset for different resolution scales.

Figure 10

Qualitative ablation results of our proposed method on the ”Bicycle” scene.

Figure 11

Qualitative ablation results of our proposed method on the ”Counter” scene.

Figure 12

Qualitative ablation results of our proposed method on the ”Garden” scene.

Figure 13

Qualitative ablation results of our proposed method on the ”Treehill” scene.

Figure 14

Qualitative ablation results of our proposed method on the ”Truck” scene.

Limitations

Since all Gaussian filtering of our proposed method relies on the pixel coverage , it can only be done after the initial splatting process when the coverage is calculated. Although the splatting of individual Gaussians are performed in parallel and does not takes more time at lower resolution, it is still a considerable overhead when rendering at a very low resolution. Even if _a very small portion of _the Gaussians are used for rendering in the end, all Gaussians still need to be splatted. This is the main reason why our rendering time is not decreased linearly as the resolution decreases.

In ourfuture work, we will look into a morelightweight criteria to filter small and large Gaussians before splatting them onto the screen to achieve an even faster rendering speed.

Conclusion

We analyzed the cause of the severe aliasing artifact and speed degradation of the existing 3D Gaussian splatting based on the general signal processing theory.

_(1) We identified thekey challenge of mitigating the aliasing for 3D Gaussian splatting lies in representing the scene with Gaussians of appropriate scale. _

_(2) Based on this observation, we propose to calculate thepixel coverage of 3D Gaussians during splatting and use this as a criteria for selective rendering. _

_(3) Gaussians that aretoo large or too small at the current rendering resolution are filtered for anti-aliasing and speed improvements. _

(4) We also proposed toinsert large Gaussians by aggregating small Gaussians during training to preserve the low frequency details and prevent part missing.

Our experiments on various datasets support the effectiveness of our algorithm in rendering quality and speed at both high and low resolution, mitigating the severe aliasing artifact of the original 3D Gaussian splatting.