Gaussian Splatting: 3D Reconstruction and Novel View Synthesis, a Review(3)
ABSTRACT
Image-based 3D reconstruction presents a complex challenge that entails deducing the three-dimensional structure of an object or environment based on input image data. Learning-driven methodologies have garnered considerable attention due to their capacity to directly infer 3D structures. This review paper is dedicated to exploring cutting-edge techniques in 3D reconstruction, particularly emphasizing the creation of novel perspectives that were previously unattainable.
DISCUSSION
Conventionally, 3D scenes have traditionally been rendered using meshes and points owing to their well-defined structure and the ability of GPU/CUDA-based rasterization to process them efficiently.
Recent advancements, such as those utilizing NeRF methods, focus on achieving continuous scene representations, which employ techniques like multilayer perceptrons optimized via volumetric ray-marching to enable novel view synthesis. Continuous representations facilitate the optimization process but require stochastic sampling during rendering, which unfortunately introduces costly noise.
Gaussian Splatting effectively bridges the gap between data density and computational efficiency in point cloud processing. By employing a 3D Gaussian representation to optimize point cloud processing, the method achieves state-of-the-art visual quality while maintaining competitive training efficiency.
Additionally, the tile-based splatting approach delivers real-time performance with excellent visual fidelity. Gaussian Splatting has achieved superior outcomes in terms of both visual quality and computational efficiency when rendering 3D scenes.
Gaussian Splatting is originally designed to manage dynamic and deformable objects through modification of its original representation. This method incorporates the 3D position, rotation, scaling factor, spherical harmonics coefficients for color and opacity into its framework.
Recent progress in this domain includes the incorporation of a sparsity-based loss to promote basis trajectory sharing, the development of a dual-domain formulation to model time-dependent residuals, and The Gaussian Shell Maps connect generator networks with 3D Gaussian rendering techniques.
Efforts have additionally addressed challenges including non-rigid tracking issues, avatar expression variations, and the efficient rendering of realistic human performance. Collectively, these advancements seek to achieve real-time rendering capabilities, optimize efficiency in processing, and deliver high-quality results in scenarios involving dynamic and deformable objects.
In addition, diffusion and Gaussian Splatting collaborate to generate 3D objects based on text-based prompts.
Diffusion models, categorized as a type of neural network, are capable of generating images from noisy inputs through the process of image degradation by gradually enhancing the quality through a sequence of increasingly clean images.
In the text-to-3D conversion pipeline, utilizing a diffusion model to generate an initial 3D point cloud based on a text description, this representation is subsequently transformed into high-gaussian spheres through advanced projection techniques like Gaussian Splatting. The rendered high-gaussian spheres collectively form the final comprehensive three-dimensional object image as captured by digital imaging systems.
Advances in this domain encompass employing structured noise as a strategy to address multi-view geometric challenges, coupled with the introduction of a variational Gaussian Splatting model designed to enhance convergence stability, and the optimization of denoising metrics to elevate the efficacy of diffusion-based priors. This endeavor is dedicated to advancing text-driven 3D generation technology towards unprecedented levels of fidelity and operational excellence.
Gaussian Splatting has achieved notable success in the synthesis of 3D digital characters for augmented reality and virtual reality applications.
This involves acquiring subjects from the smallest possible number of viewpoints and creating 3D models. The technique has been employed to construct detailed models of human anatomy articulation, joint angles, and other relevant parameters, enabling the creation of highly expressive and controllable digital representations.
Advancements in this area include innovative techniques for the acquisition of high-frequency facial details, the maintenance of exaggerated facial expressions, and the efficient deformation of avatar models.
Additionally, hybrid models have been developed, incorporating both expressive explicit representations with hidden learnable latent features to enable expression-based final color and opacity values.
These advancements target enhancing the quality of generated 3D models' geometry and texture, serving the growing need for realistic and controllable avatars in AR/VR environments.
Gaussian projection also demonstrates versatile utilization in the field of SLAM, effectively enabling real-time tracking and mapping tasks on GPU architectures.
Using a 3D Gaussian description and a differentiable projection of the splatting rasterization pipeline, it can rapidly produce photorealistic rendering for both real-world scenarios and synthetic datasets.
This technique extends its application to cover mesh extraction and physics-based simulations, enabling the modeling of mechanical properties without requiring explicit object meshing.
Recent advancements in the continuum mechanics field and partial differential equations (PDEs) have allowed for the development of Gaussian kernels, enhancing motion generation.
Significantly, optimizations include efficient data structures such as OpenVDB, regularization terms aligned with alignment, physics-inspired terms aimed at minimizing errors, which collectively improve the system’s overall efficiency and accuracy.
Prior studies have focused on compression techniques, and enhancing the rendering performance of Gaussian Splatting has been a significant area of research.
COMPARISON
Gaussian Splatting represents the most suitable technology for real-time rendering and also dynamic scene representation.
Occupancy network are not at all tailored for NVS use case.
Photogrammetry is best suited for generating highly precise, detailed, and lifelike models that inherently incorporate contextual understanding.
NeRFs擅长于生成独特的视角并提供逼真的照明效果，在艺术创作与科学实验中展现出卓越的创造力与应对复杂场景的能力。
Gaussian Splatting excels in its real-time rendering features and interactive exploration techniques, making it well-suited for dynamic applications.
Distinct approaches each serve their own specialty, working together to provide a comprehensive set of techniques for 3D reconstruction and visualization.
CHALLENGES AND LIMITATIONS
Computational complexity
Gaussian Splatting involves calculating the Gaussian values at each pixel coordinate, a process that demands significant computational resources, particularly when handling a massive number of points or particles.
Memory usage
Keeping track of intermediate results for Gaussian Splatting, which involve the weighted contributions of each point to neighboring pixels, requires a substantial amount of memory storage.
Edge artifacts
Gaussian Splatting generates negative artifacts adjacent to image edges or regions with high contrast, including phenomena such as ringing and blurring effects.
Performance vs. accuracy trade-off
Achieving high-quality outcomes could involve applying large convolution kernels or utilizing a combination of Gaussian functions across each pixel, which in turn affects the system’s performance metrics.
Integration with other rendering techniques
Combining Gaussian Splatting with other techniques such as shadow mapping or ambient occlusion while ensuring performance and visual coherence can be complex.
FUTURE DIRECTIONS
Real-time 3D reconstruction techniques will provide a range of functionalities in computer graphics and related disciplines, including engaging interactive exploration of 3D scenes or models at real-time rates, enabling direct manipulation of viewpoints and objects with instant visual feedback.
It will also support real-time rendering of complex and dynamic visualizations, such as objects in motion or varying surroundings, at runtime, increasing visual fidelity and engagement.
Real-time 3D reconstruction is effectively applied in simulation and training environments, offering realistic visual feedback for virtual scenarios across sectors such as automotive, aerospace, and medicine.
It will also facilitate real-time renderings of immersive AR and VR experiences, where users can fully immerse themselves in virtual objects or environments.
Comprehensive real-time Gaussian Splatting imparts computational efficiency to diverse applications within the domains of computer graphics, visualization, simulation, and immersive technologies. It also enhances user interaction and visual fidelity.
CONCLUSION
this paper explores a variety of functional and application-oriented features associated with Gaussian Splatting in the context of three-dimensional reconstruction tasks and innovative approaches to novel view synthesis.
Comprising a diverse range of topics including dynamic modeling of shape deformation、motion analysis、non-rigid objects、expression variation or emotion diversity、text generation through a diffusion process、noise reduction techniques、optimization-driven approaches、avatar models for interactive content、animatable media for dynamic experiences、head-based modeling for realistic representations、integrated localization and mapping solutions、mesh generation with physical simulation capabilities以及 advanced optimization algorithms to enhance efficiency and effectiveness in various applications
Gaussian Splatting holds significant potential in its innovative applications across various domains such as visual computing virtual and augmented reality robotic technologies visual media production automotive engineering e-commerce sector sustainability research and Aeronautics innovations.
It is crucial to highlight that Gaussian Splatting does exhibit significant shortcomings in its ability to achieve photorealism relative to other methods, including NeRFs among other methods.
Additionally, challenges including overfitting, computational resource consumption, and rendering quality limitations should be considered.