Texture-GS: Disentangling the Geometry and Texture for 3D Gaussian Splatting Editing

Abstract

3DGS, emerging as a groundbreaking approach, has drawn increasing attention for its capabilities of high-fidelity reconstruction and real-time rendering. However, it couples the appearance and geometry of the scene within the Gaussian attributes , which hinders the flexibility of editing operations(such as texture swapping).

To address this issue, we propose a novel approach, namely Texture-GS , to disentangle the appearance from the geometry by representing it as a 2D texture mapped onto the 3D surface , thereby facilitating appearance editing.

Technically, the disentanglement is achieved by our proposed texture mapping module , which consists of a UV mapping MLP to learn the UV coordinates for the 3D Gaussian centers, a local Taylor expansion of the MLP to efficiently approximate the UV coordinates for the ray-Gaussian intersections , and a learnable texture to capture the fine-grained appearance.

Extensive experiments on the DTU dataset demonstrate that our method not only facilitates high-fidelity appearance editing but also achieves real-time rendering on consumer-level devices, e.g. a single RTX 2080 Ti GPU.

Figure

Figure 1

Texture swapping with our method.

We propose to disentangle the appearance from the geometry for 3DGS, thereby facilitating real-time appearance editing(such as texture swapping).

Figure 2

Comparison with the straightforward solution.

The straightforward solution fails to generate a reasonable and continuous texture, while our method can reconstruct a high-quality texture by considering the intersection of 3D Gaussians and each ray.

Figure 3

Visual comparison with previous SOTA editing methods

Figure 4

Visual comparison of our method with different numbers of 3D Gaussians.

Figure 5

Visualization of texture swapping results of our method.

Figure 6

Visual comparison of our method with NeuTex on texture swapping.

Figure 7

Visualization of texture painting results of our method.

Figure 8

Ablation study of intersection-based UV mapping.

Figure 9

Ablation study of the per-Gaussian SH coefficients on the DTU scene.

Figure 10

Ablation study of the pruning strategy during 3D Gaussian optimization.

Figure 11

Shadow-preserving texture swapping.

Figure 12

Visual comparison with 3DGS under the same number of 3D Gaussians.

Figure 13

Failure cases.

Due to the limited representational power of the UV mapping MLP, our method fails to learn a uniform and reasonable texture space for objects that have thin plates or holes , thereby hindering downstream applications.

Figure 14

More visual results for texture swapping with Texture-GS.

Limitations

The edges of the chessboard texture are not distinctly clear in Fig. 10. We argue that theblurring issue primarily stems from the**inaccurate orientations** of 3D Gaussians , which leads to incorrect UV coordinates for color fetching.

In addition, we define the texture space as a unit sphere , which is ill-suited to represent multiple objects or outdoor scenes. Representing the UV mapping with multiple charts(such as Nuvo), can address this problem, which is not the focus in this paper.

Conclusion

We present a novel method, namely Texture-GS, which disentangles the appearance and geometry for 3DGS to facilitate various appearance editing operations(such as texture swapping).

To achieve this, we incorporate atexture mapping module into 3DGS, which consists of a UV mapping MLP that projects 3D points into 2D UV space, **a local Taylor expansio n **for efficient UV mapping and a learnable 2D texture to capture the appearance of 3D scenes.

Experiments demonstrate our method is not only capable of reconstructing high-fidelity textures from multi-view images, but also enables various real-time texture editing application. We hope that this work will provide researchers with a more profound comprehension of the relationship between 3D Gaussians and meshes, serving as a launchpad for further exploration.