Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images

Abstract

In this paper, we propose a novel method for improving Gaussian splatting by using a small set of reference images while preventing overfitting.

Representing a 3D scene using numerous Gaussian splats achieves highly accurate results. However, this approach tends to exhibit overfitting tendencies when the number of training images is limited.

This challenge is addressed by presenting high-density depth mapping data as a geometric reference framework to mitigate overfitting.

We estimated the depth map by employing a trained-in-advance monocular depth estimation model to adjust scale parameters and offset parameters via sparse COLMAP feature points.

The adjusted depth plays a key role in the color optimization of 3DGS, effectively eliminating floating-point artifacts while ensuring compliance with geometric constraints.

We evaluate our approach across diverse datasets, including the NeRF-LLFF collection, which features a range of image configurations. Our technique exhibits superior geometric performance when compared to conventional methods reliant solely on image data.

Figure

Figure 1

The efficacy of depth regularization in a few-shot setting.

In this study, we maximize the efficiency of Gaussian splats by using a limited number of images to avoid overfitting through geometry guidance estimated from these images.

Please note that we utilized onlytwo images to create this 3D scene.

Figure 2

We employ 3DGD based on detailed depth information aligned with point clouds generated by the COLMAP system.

Our model, by utilizing depth maps to manipulate or adjust the geometric structure of 3D scenes, accurately reconstructs 3D scenes with minimal image input.

Figure 3

Qualitative comparison in NeRF-LLFF dataset.

The differences between 3DGS and our approach are effectively demonstrated across multiple viewpoints.

Driven primarily by color loss, 3DGS encountered challenges in achieving expected geometric configurations. Through our method, we were able to establish credible geometric structures guided by depth information consistently, leading to outstanding reconstruction results.

Figure 4

Details in cropped patches.

Our approach generates more accurate reconstruction outcomes than 3DGS by exploiting additional geometric cues. Our method generates stable geometry with superior performance in reconstruction quality compared to 3DGS.

Figure 5

Example results utilizing pseudo-GT depth (oracle).

The capability to attain precise depth measurements allows for the creation of highly detailed 3D models, even when relying on a small set of images. Fine details can be perceived in both RGB and depth channels.

Limitation and Future Work

Our approach successfully established the effectiveness of Gaussian splatting optimization in low data scenarios through depth guidance, yet it is characterized by notable shortcomings.

First and foremost, this method greatly depends on the estimating capability of the monocular depth estimation model.

Moreover, this model’s depth estimation performance can vary based on the learned data domain , consequently affecting the performance of Gaussian splatting optimization.

Furthermore, depending on the estimated depth being aligned with COLMAP points implies a reliance on COLMAP’s performance, which thus makes it unable to address textureless plains or challenging surfaces where COLMAP might falter.

Identify as a potential area for future research the improvement of 3D scene reconstruction using interconnected depth estimates instead of COLMAP points. Additionally, investigating techniques to enforce geometric consistency across diverse datasets, especially in regions like the sky where depth recovery may present challenges, presents another promising direction for future research.

Conclusion

In this work, we present Depth-Regularized Optimization aimed at advancing 3DGS within the domain of Few-Shot Images. This model is designed to learn 3DGS effectively from a limited dataset.

The splats are enforced by our model through the use of depth, highlighting their effectiveness in geometric guidance.

Obtaining dense depth information, we utilize a monocular depth estimation model to modify the depth scale based on SfM-derived points.

We assessed the effectiveness of our proposed depth loss measure, unsupervised smoothness constraint mechanism, and early termination criterion within the NeRF-LLFF dataset. Our method demonstrated superior performance in few-shot learning framework, generating plausible geometric structures.

Finally, we demonstrated through additional experiments that improved depth and initialization points significantly enhance the performance of 3DGS-based reconstruction.