GaussianPro: 3D Gaussian Splatting with Progressive Propagation(Related Work)
Multi-view Stereo
MVS aims to reconstruct a 3D model from a collection of posed images, which can be further combined with traditional rendering algorithms to generate novel views.
Traditional methods explicitly establish pixel correspondences between images based on hand-crafted image features and then optimize the 3D structure to achieve the best pixel correspondences among images.
Learning-based MVS methods implicitly build multi-view correspondences with learnable features and regress depth or 3D volume based on the features in an end-to-end framework.
GaussianPro draws inspiration from depth optimization in MVS to improve the geometry of the Gaussians, thereby achieving better rendering results.
Neural Radiance Field
NeRF combines deep learning techniques with the 3D volumetric representation, transforming a 3D scene into a learnable continuous density field.
Utilizingray marching in volume rendering, NeRF is able to achieve highquality novel view synthesis without explicit modeling of the 3D scene and illumination.
To further improve the rendering quality, some approaches directly improve the point sampling strategy in ray marching for more accurate modeling of the volume rendering process.
Others improve rendering by reparameterizing the scene to generate more compact scene representation and easier learning process.
Additionally, regularization terms could be introduced to constrain the scene representation towards a closer approximation of real geometry.
Despite these advancements, NeRF still incurs high computational costs during rendering.
Since NeRF employs MLPs to represent the scene, the computation and optimization of any point in the scene are dependent on the entire MLP.
Many works propose novel scene representations to accelerate rendering.
They replaceMLPs with sparse voxels , hash tables , or triplane , allowing the computation and optimization of each point to be localized to the corresponding local region of the scene.
Although these methods significantly improve rendering speed, real-time rendering is still challenging due to the inherent ray marching strategy in volume rendering.
3D Gaussian Splatting
3DGS employs a spatting-based rasterization approach to project anisotropic 3D Gaussians onto a 2D screen.
It computes the pixel’s color by performing depth sorting and α-blending on the projected 2D Gaussians, which avoids the sophisticated sampling strategy of ray marching and achieves real-time rendering.
Some concurrent works have made improvements to 3DGS.
Firstly, 3DGS is sensitive to sampling frequency , i.e., changing the camera’s focal length or camera distance could result inrendering artifacts. These artifacts are addressed by introducing low-pass filtering or multiscale Gaussian representations.
Additionally, 3DGS excessively grows Gaussians without explicitly constraining the scene’s real geometric structure , resulting in numerous redundant Gaussians and significant memory consumption. Some methods evaluate the __contribution __of Gaussians to rendering by their scales or calculating theirvisibility in views, forcing the removal of Gaussians with small contributions.
Others compress the storage of Gaussian attributes by quantization technique or interpolating Gaussian attributes from structured grid features . Although these methods significantly reduce the storage overhead of Gaussians, they do not explicitly __constrain the geometry __of the Gaussians.
3DGS could grow in locations far from the real surfaces to fit different training views, resulting in redundancy and a decrease in rendering quality for new viewpoints.
GaussianPro considers the planar prior in the scene, explicitly constraining the growth of Gaussians close to the real surfaces. This approach enables Gaussians to better fit the real geometry of the scene, achieving improved rendering and more compact representation.
