3D Gaussian Splatting in Robotics: A Survey(3)

Abstract

Maintaining detailed 3D environmental representations has long served as a key objective within the robotics research community.

Previously prevalent NeRF representations have been widely used based on an implicit coordinate system. The recent rise of 3DGS has demonstrated significant potential in an explicit representation of radiance fields.

By employing 3D Gaussian primitives for detailed 3D scene modeling and enabling differentiable rendering, 3DGS demonstrates notable superiority compared to other approaches in real-time rendering and photo-realistic performance; these benefits are highly advantageous for robotic applications.

FUTURE RESEARCH

Robust Tracking

Existing 3DGS-based SLAM methods, although showing high accuracy in dense mapping, typically struggle to attain precise and reliable tracking, particularly when focusing on challenging real-world environments.

This limitation in current 3DGS-based SLAM systems is due to their reliance on directly using RGB information of image for pose optimization.

Significantly, the dependence on the qualities and texture information of the images is crucial.

Despite their widespread use in real-world robotic systems, image quality remains a significant challenge due to camera motion blur , which can seriously impair the effectiveness of 3DGS-based SLAM algorithms.

To date, there are certain scenes lacking detailed texture details, such as the sky or wall surfaces, which make it challenging to accurately determine the object's pose.

The following outlines specific steps to enhance the robustness of tracking.

Camera motion blur

The primary cause of camera motion blur stems from rapid robot movements coupled with sluggish camera shutter speeds, resulting in blurred imagery. While various deblurring techniques have been developed and implemented within SLAM systems, none have successfully achieved direct conversion of captured blurred imagery into sharp results.

Instead, they synthesize motion blur effects by capturing and blending virtual sharp images while the camera is capturing to create blurry images.

Such synthesized blurry images are subsequently employed to formulate loss in conjunction with observed blurry images within Gaussian optimization, thus guaranteeing that the reconstructed scene achieves deblurring.

However, such methods are found to be inadequate in addressing the damage to image quality in the observed images resulting from motion blur. This failure significantly impacts tracking performance, which heavily relies on maintaining high image quality for pose estimation.

A promising research direction is to exploit the advantages of 3DGS representation(including geometric information and spatial distribution), for tracking purposes. This approach minimizes reliance on image quality.

Limited texture information

Within real-world scenarios, there are certain boundary conditions where environmental texture information is lacking, resulting in inadequate constraints for pose optimization relying solely on image quality.

Although some 3DGS-based SLAM methods have incorporated multi-sensor fusion of traditional SLAM as Odometry for tracking, these methods struggle when traditional SLAM is unable to manage complex corner cases.

A promising research direction lies in integratingraw sensor data from multiple sources (including IMU, wheel encoders, and LiDAR) with 3D Gaussian representations to establish robust constraints for pose optimization.

This approach not only leverages the spatial structural information and dense scene representation offered by 3DGS, but also incorporates diverse constraints derived from multi-sensor data.

Lifelong Mapping and Localization

Current 3DGS methods are mainly dedicated to short-term reconstruction and localization tasks. However, in most real-world scenarios, the environment undergoes continuous changes over time.

A prebuilt map that is not able to account for such modifications tends to become inaccurate or dependable over time.

Thus, it is essential for maintaining an environmental model to enable prolonged operation and navigation of robots.

改写说明

Given this, it is evident that continuous 3D GS-based dense scene understanding and position estimation. represents a promising research frontier for advancing autonomous systems.

Due to 3DGS being an explicitly detailed and compact representation, it has been demonstrated that the continuous updating and refining process of the Gaussian map can be effectively implemented through targeted modifications to its underlying Gaussian primitives.

Furthermore, we believe that the inconsistencies in the Gaussian map resulting fromevolutionary modifications can be improved upon by exploiting theinternal regulations governing the interactions among Gaussian primitives.

Therefore, by exploiting theclear mathematical formulations and built-in limitations of Gaussian primitives, a self-consistent positioning solution can be achieved throughout the entire operational lifecycle.

Large-scale Relocalization

Within the domain of robotic applications, it is imperative that robots must reposition themselves when they encounter a pre-established map. Nevertheless, existing 3DGS-based relocalization methods typically rely on having a rough initial pose or are limited to achieving relocalization in narrow indoor spaces.

These methods find difficulty in performing relocalization for extensive outdoor environments without an initial pose estimate.

Unfortunately, it is difficult to acquire a rough initial estimate for relocalization in practical robotic systems.

因此，在大规模重定位中无需初始姿态是一种具有重要意义的研究方向。

We regard the construction of a submap index library or descriptor, based on the 3DGS representation, as providing support for coarse pose regression.

In addition, the coarse pose can be refined through a registration process that leverages geometric and appearance features within the 3DGS representation.

Sim-to-Real Manipulation

Collecting real-world manipulation datasets is a complex task, which often results in insufficient data for effectively training robotic systems to perform precise grasps in realistic environments. Consequently, many grasping strategies are developed and refined primarily through extensive training and testing within controlled simulation environments before being implemented in actual applications.

Despite the extensive study of 3DGS-based simulation-to-reality approaches, these methods still exhibit shortcomings in their generalization capability.

Specifically, this approach relies heavily on scene-specific training, making it difficult for the system to generalize and transfer the acquired knowledge across comparable tasks.

Consequently, this method still requires a large number of real-world datasets for training.

Additionally, the variations in physical characteristics between simulated versus real-world settings can lead to significant changes in the patterns of training datasets used for manipulation tasks. Such variations may ultimately result in entirely different operational strategies.

However, despite the existing method, it is limited to physical characteristics of real-life situations.

Given that, one promising area of research involves incorporating the integration of uncertainty and environmental features into the 3DGS representation to improve the model's ability to generalize and accurately capture properties.