SuGaR: Surface-Aligned GS for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering

Abstract

We propose a method to allow precise and extremely fast mesh extraction from 3DGS. Gaussian Splatting has recently become very popular as it yields realistic rendering while being significantly faster to train than NeRFs. It is however challenging toextract a mesh from the millions of tiny 3D Gaussians as these Gaussians tend to be unorganized after optimization and no method has been proposed so far.

Our first key contribution is a regularization term that encourages the Gaussians to align well with the surface of the scene.

We then introduce a method that exploits this alignment to extract a mesh from the Gaussians using Poisson reconstruction , which is fast, scalable, and preserves details , in contrast to the Marching Cubes algorithm usually applied to extract meshes from Neural SDFs.

Finally, we introduce an optional refinement strategy that binds Gaussians to the surface of the mesh , and jointly optimizes these Gaussians and the mesh through Gaussian splatting rendering. This enables** easy editing, sculpting, animating, and relighting** of the Gaussians by manipulating the mesh instead of the Gaussians themselves.

Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the SOTA method on SDFs, while providing a better rendering quality.

Figure

Figure 1

We introduce a method that extracts accurate and editable meshes from 3DGS representations within minutes on a single GPU.

The meshes can be edited, animated, composited, etc. with very realistic Gaussian Splatting rendering, offering new possibilities for Computer Graphics.

Note for example that we changed the posture of the robot between the captured scene on the bottom left and the composited scene on the right. The supplementary material provides more examples, including a video illustrating our results.

Figure 2

Our algorithm can extract a highly detailed mesh from any 3DGS scene within minutes on a single GPU

top Renderings of our meshes without texture ,

bottom Renderings of the meshes with bound Gaussians.

Figure 3

Extracting a mesh from Gaussians.

Without regularization, the Gaussians have no special arrangement after optimization, which makes extracting a mesh very difficult.

Without our regularization term, Marching Cubes fail to extract an acceptable mesh.

With our regularization term, Marching Cubes recover an extremely noisy mesh even with a very fine 3D grid. Our scalable extraction method obtains a mesh even without our regularization term. Still, the mesh is noisy. By contrast, our full method succeeds in reconstructing an accurate mesh very efficiently.

Figure 4

Examples of**(a)** renderings and (b) reconstructed meshes with SuGaR. The (c) normal maps help visualize the geometry.

Figure 5

Efficiently estimating of the SDF of the surface generated from Gaussians.

We render depth maps of the Gaussians, sample points in the viewpoint according to the distribution of the Gaussians.

Value is taken as the 3D distance between and the intersection between the line of sight for p and the depth map.

Figure 6

Sampling points on a level set for Poisson reconstruction.

(a) We sample points on the depth maps of the Gaussians and refine the point locations to move the points on the level set.

(b) Comparison between the extracted mesh without (left) and with (right) our refinement step. Since splatted depth maps are not exact, using directly the depth points for reconstruction usually results in a large amount of noise and missing details.

Figure 7

Joint refinement of mesh and Gaussians.

(a) We bind Gaussians to the triangles of the mesh. Depending on the number of triangles in the scene, we bind a different number of Gaussians per triangle, with predefined barycentric coordinates.

(b) Mesh before and after joint refinement.

Figure 8

Refined SuGaR renderings with different numbers of refinement iterations.

(a)2,000 iterations are usually enough to obtain high quality rendering, since the extracted mesh “textured” with surface Gaussians is already an excellent initialization for optimizing the model. (b-c)further refinement helps the Gaussians to capture texturing details and reconstruct extremely thin geometry that is finer that the resolution of the mesh, such as the spokes of the bicycle.

Figure 9

SuGaR renderings with different vertices.

Even with low-poly meshes, the 3D Gaussians bound to the mesh produce high quality renderings. Moreover, low-poly meshes help to better regularize the surface.20w，1,00w

Figure 10

Qualitative comparison between (top) a traditional UV texture optimized from training images, and (bottom) the bound surface Gaussians.

Even though high resolution UV textures have good quality and can be rendered with our meshes using any traditional software, using 3D Gaussians bound to the surface of the mesh greatly improves the rendering quality. Meshes in these images have 200,000 vertices only.

Conclusion

We proposed a very fast algorithm to obtain an accurate 3D triangle mesh for a scene via Gaussian Splatting. Moreover, by combining meshing and Gaussian Splatting , we make possible intuitive manipulation of the captured scenes and realistic rendering, offering new possibilities for creators.