GS-ROR: 3D Gaussian Splatting for Reflective Object Relighting via SDF Priors

Abstract

3DGS has shown a powerful capability for novel view synthesis due to its detailed expressive ability and highly efficient rendering speed. Unfortunately, creating relightable 3D assets with 3DGS is still problematic, particularly for reflective objects , as its discontinuous representation raises difficulties in constraining geometries.

Inspired by previous works, the signed distance field (SDF) can serve as an effective way for geometry regularization. However, a direct incorporation between Gaussians and SDF significantly slows training.

To this end, we propose GS-ROR for reflective objects relighting with 3DGS aided by SDF priors.

At the core of our method is the mutual supervision of the depth and normal between deferred Gaussians and SDF, which avoids the expensive volume rendering of SDF. Thanks to this mutual supervision, the learned deferred Gaussians are well-constrained with a minimal time cost.

As the Gaussians are rendered in a deferred shading mode , while the alpha-blended Gaussians are smooth, individual Gaussians may still be outliers, yielding floater artifacts. Therefore, we further introduce an SDF-aware pruning strategy to remove Gaussian outliers, which are located distant from the surface defined by SDF , avoiding the floater issue. Consequently, our method outperforms the existing Gaussian-based inverse rendering methods in terms of relighting quality.

Our method also exhibits competitive relighting quality compared to NeRF-based methods with at most 25% of training time and allows rendering at 200+ frames per second on an RTX4090.

Figure

Figure 1

We present an SDF-aided Gaussian Splatting framework for Reflective Object Relighting (GS-ROR) from multi-view images.

We show five relighting results with reflective highlights (left) and their normal estimation (right), including Potion and Tbell from NeRO, Helmet and Toaster from Ref-NeRF, andQilin from NeILF++, where Qilin is from the real scene.

Our method demonstrates a robust geometry reconstruction for reflective surfaces and faithful material decomposition , leading to photo-realistic and real-time reflective object relighting.

Figure 2

The geometry from Gaussian is under-constrained and thus erroneous, while it is much better after utilizing the priors from the SDF.

Figure 3

Overview of our framework.

The architecture of our proposed method consists of two geometry representations (i.e., Gaussian primitive and TensoSDF).

In the deferred Gaussian pipeline, the shading parameters (i.e., albedo , roughness and metallicity ),normal and depth are projected to the image plane and alpha blended.

The pixel color is calculated using the split-sum approximation and supervised by ground truth color .

In the TensoSDF, we sample rays originated from camera center and view direction and query the SDF value and gradient for each point along the ray .

The normal and depth are obtained via volume rendering , which is supervised mutually with the normal the depth from Gaussians.

Note no color network is used in the SDF part, and only the geometry attributes are volume rendered.

Figure 4

Two Gaussians with a minor normal difference are overlapped to model an opaque surface.

In the forward rendering, the BRDF values are computed via their own normal and are then alpha blended to form the final rendering, which is equivalent to a broader BRDF lobe , leading to a blurry rendering, eventually.

In contrast, in the deferred shading, the BRDF is computed via thedeferred normal , maintaining the sharpness of reflective objects.

Figure 5

In this example, the floaters can still be shown in the testing view, even if the mask loss was applied during training.

The main reason is that althoughthe floaters are shown outside the mask region in the testing view, they are within the mask region in the training views. Therefore, they can not be masked out by the mask loss.

Figure 6

The illustration of SDF-aware pruning.

We define anarrowing threshold , which is adjusted automatically around the zero-level set.

The Gaussians out of the threshold will be pruned.

This pruning operation ensures all Gaussians are near the surface and avoids the floaters.

Figure 7

Ablation of several key components in our method, including the deferred shading, SDF supervision and pruning. The insets are shown in blue, and some floaters are shown in green.

Figure 8

Our method preserves the details that are lost in the SDF-based method (i.e., NeRO), revealing the benefits of the Gaussian representation.

Figure 9

The comparison between the result using Cycle path-tracing renderer in Blender and the one using split-sum approximation in DiffRast. The splitsum result is blurry with extremely low roughness.

Discussion and limitations

Our model improves the relighting quality compared to other Gaussianbased methods. However, some issues remain to be solved.

The SDF representation used in our framework is used as a prior, and it can not be used for exporting a faithful mesh due to itslow resolution.

Besides, we obverse the split-sum approximation in DiffRast causes blurry renderings for extremely low-roughness surfaces.

We render a mirror-like plane using the Cycle path-tracing renderer in Blender and the split-sum approximation in the DiffRast. The comparison is in Fig. 9, and the split-sum result is blurry.

We believe replacing it with a more accurate rendering method will improve the relighting quality. Last, we only consider direct lighting in our framework, and introducing indirect illumination will benefit the relighting quality. We leave it for future work.

Conclusion

In this paper, we present a novel framework for real-time reflective object inverse rendering.

We design an SDF-aided Gaussian splatting framework, using the mutual supervision of the depth and normal between deferred Gaussians and SDF to improve the quality of geometry from Gaussians.

Besides, we propose an SDF-aware pruning strategy with an automatically adjusted threshold, regularizing the position of Gaussian and avoiding the floater artifact.

Consequently, our method outperforms the existing Gaussian-based inverse rendering methods without losing efficiency. It is competitive with the NeRF-based methods in terms of quality, with much less training and rendering time.

In future work, extending our framework to complex scenarios with multiples objects is a potential direction.