A Survey on 3D Gaussian Splatting(1)

Abstract

3DGS has recently emerged as a transformative technique in the explicit radiance field and computer graphics landscape. This innovative approach, characterized by the utilization of millions of 3D Gaussians , represents a significant departure from the neural radiance field (NeRF) methodologies, which predominantly use implicit, coordinate-based models to map spatial coordinates to pixel values.

3DGS, with its explicit scene representations and differentiable rendering algorithms, not only promises real-time rendering capabilities but also introduces unprecedented levels of control and editability. This positions 3DGS as a potential game-changer for the next generation of 3D reconstruction and representation.

Figure

Figure 1

The number of papers on 3D GS is increasing every month.

Figure 2

Structure of the overall review.

Figure 3

An illustration of the forward process of 3DGS.

(a) The splatting step projects 3D Gaussians into image space.

(b) 3DGS divides the image into multiple non-overlapping patches , i.e., tiles.

(c) 3DGS replicates the Gaussians which cover several tiles , assigning each copy an identifier , i.e., a tile ID.

(d) Byrendering the sorted Gaussians, we can obtain all pixels within the tile.

Note that the computational workflows for pixels and tiles are independent and can be donein parallel.

FUTURE RESEARCH DIRECTIONS

Data-Efficient 3DGS Solutions

The generation of novel views and the reconstruction of scenes from limited data points are of significant interest, particularly for their potential to enhance realism and user experience with minimal input.

Recent advances have explored the use of depth information, dense probability distributions, and pixel-to-Gaussian mapping to facilitate this capability. However, there remains an urgent need for further exploration in this domain.

In addition, a notable issue of 3DGS is the emergence of artifacts in areas with insufficient observational data.

This challenge is a prevalent limitation in radiance field rendering, where sparse data often leads to inaccuracies in reconstruction.

Consequently, the development of novel methods for data interpolationorintegration in these sparse regions represents a promising avenue for future research.

Memory-Efficient 3DGS Solutions

While 3D GS demonstrates remarkable capabilities, its scalability poses significant challenges, particularly when juxtaposed with NeRF-based methods. The latter benefits from the simplicity of storing merely the parameters of a learned MLP.

This scalability issue becomes increasingly acute in the context oflarge-scale scene management, where the computational and memory demands escalate substantially.

Consequently, there is an urgent need to optimize memory utilization during both the training phase and in the storage of the model.

Exploring more efficient data structures and investigating advanced compression techniques represent promising avenues to address these limitations.

Advanced Rendering Algorithms

The current rendering pipeline of 3DGS is 4straightforward and can be further optimized.

For instance, thesimple visibility algorithm may lead to cause a drastic switch in depth/blending order of Gaussians.

This underscores a significant opportunity for future research: the implementation of more advanced rendering algorithms.

These improved methodologies should aim to more accurately simulate the intricate interplay of light and material properties within a given scene.

A promising approach could involve the assimilation and adaptation of established principles from traditional computer graphics into the specific context of 3DGS.

Noteworthy in this regard is the ongoing effort to integrate enhanced rendering techniques or hybrid models into current computational frameworks of 3DGS.

Furthermore, the exploration of inverse rendering and its applications presents a fertile ground for investigation.

Optimization and Regularization

The anisotropic Gaussians , while beneficial for representing complex geometries, can create undesirable visualartifacts.

For example,thoselarge 3D Gaussians , especially in regions with view-dependent appearance, can cause popping artifacts , where visual elements abruptly appear or disappear , breaking the immersion.

There is considerable potential for exploration in the r egularization and optimization of 3DGS.

Introducing antialiasing could mitigate the abrupt changes in depth and blending order of Gaussians.

Enhancements in optimization algorithms might better control Gaussians in space and beyond.

Further, incorporatingregularization into the optimization process may accelerate convergence , smooth visual noise, or improve image quality.

In addition, sucha large number of hyper-parameters affects the generalization of 3DGS, which is in dire need of a solution.

3D Gaussians in Mesh Reconstruction

The potential of 3DGS in mesh reconstruction and its position in thespectrum of volumetric and surface representations is yet to be fully explored.

Researching howGaussian primitives can be adapted formesh reconstruction tasks is urgently needed.

This exploration could bridge the gap between volumetric rendering and traditional surface-based methods , offering insights into new rendering techniques and applications.

There are some early explorations about mesh extraction and reconstruction based on 3D Gaussians.

Empowering 3D GS with More Possibilities

Despite the significant potential of 3DGS, the full scope of applications for 3DGS remains largely untapped.

A promising avenue for exploration involves augmenting 3D Gaussians with __additional attributes_(_such as linguistic and physical properties ) , tailored for specific applications.

Moreover, recent studies have begun to unveil the capability of 3DGS in several domains(camera pose estimation, the capture of hand-object interactions, and the quantification of uncertainty). These preliminary findings highlight a significant opportunity for interdisciplinary scholars to explore 3DGS further.