Mesh-based Gaussian Splatting for Real-time Large-scale Deformation
Abstract
Neural implicit representations, including Neural Distance Fields and Neural Radiance Fields , have demonstrated significant capabilities for reconstructing surfaces with complicated geometry and topology , and generating novel views of a scene. Nevertheless, it is challenging for users to directly deform or manipulate these implicit representations with large deformations in the real-time fashion. 3DGS has recently become a promising method with explicit geometry for representing static scenes and facilitating high-quality and real-time synthesis of novel views. However, it cannot be easilydeformed due to the use of discrete Gaussians and lack of explicit topology.
To address this, we develop a novel GS-based method that enables interactive deformation. Our key idea is to design an innovative mesh-based GS representation , which is integrated into Gaussian learning and manipulation.
3D Gaussians are defined over an explicit mesh, and they are bound with each other : the rendering of 3D Gaussians guides the mesh face split for adaptive refinement , and the mesh face split directs the splitting of 3D Gaussians.
Moreover, the explicit mesh constraints help regularize the Gaussian distribution , suppressing poor-quality Gaussians (e.g. , misaligned Gaussians, long-narrow shaped Gaussians), thus enhancing visual quality and avoiding artifacts during deformation.
Based on this representation, we further introduce a large-scale Gaussian deformation technique to enable deformable GS, which alters theparameters of 3D Gaussians according to the manipulation of the associated mesh.
Our method benefits from existing mesh deformation datasets for more realistic data-driven Gaussian deformation. Extensive experiments show that our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate (65 FPS on average on a single commodity GPU).
Figure
Figure 1
Given a set of multi-view images of an object, we reconstructs the object with the proposed Mesh-based GS representation, involving both 3D Gaussians and an associated mesh.
the mesh is adaptive refined along with Gaussian splitting, and also served as effective regularization.
As a result, our method achieves higher-quality novel view synthesis , compared with 3DGS. our 3D Gaussian deformation method produces high quality deformation results in the real-time manner with large scale deformations.
Figure 2
An overview of our pipeline.
Our mesh-based GS representation is specifically designed for Gaussian deformation.
Given a set of calibrated images, we first reconstruct an explicit mesh using the explicit geometry prior to initialize the Gaussian. During the learning, the explicit mesh guides the Gaussian learning in two strategies according to the explicit mesh: a) Face Split; b) Normal Guidance.
Also, we introduce a regularization loss
to constrain the scale of Gaussians to prevent the abnormally shaped Gaussians with extreme anisotropy.
Furthermore, the mesh-based deformation incorporates our mesh-based GS representatio n to achieve real-time deformation on 3DGS. The deformations on the explicit mesh (deformation gradients) from the user’s controls drive the parameters of Gaussians and produce the deformed 3DGS for novel view rendering.
Overall, our pipeline not only achieves accurate and realistic rendering from novel views but also supports effortless and real-time deformation of 3DGS.
Figure 3
Comparison with the alternative methods.
We show comparisons of ours to previous methods and the editing results from the novel views. There are MIC from NeRF-Synthetic, CUBIOD,D RESS and DEER captured by ourselves. We have highlighted the difference with different color boxes for different views. From the results, we can see that our method successfully preserves the high-frequency details after large-scale deformation.
Figure 4
More 3DGS deformation results.
It illustrates our proposed methods of synthesizing three novel views after making modifications using 8 examples, including BUTTERFLY,SOFA,CUBOID from the SketchFab ,GIRAFFE captured by ourself , LEGO,FICUS,CHAIR from NeRF-Synthetic dataset ,REAL-CAPTURED HUMAN from THuman3.0 Dataset.
Each example consists of 2 edits. It is clearly shown that our results are more realistic and high-fidelity from novel view rendering, as well as the various deformations.
Figure 5
Ablations on Face Split operation and regularization 
. We perform the qualitative comparison on our two ablated versions: w/o Face Split and w/o 
. The two cases illustrate that our full method can achieve the best results when large scale deformation appears.
Especially for 
, it becomes more evident that there are a greater number of Gaussians with irrational shapes when Lr is not present. Disabling Face Split may result in blurry artifacts visually.
Figure 6
Ablation of Normal Guidance on Novel View Synthesis. we perform the qualitative evaluation on Normal Guidance on Novel View Synthesis task. It is more clear that Normal Guidance can enhance the high-frequency details and complex structures significantly from novel view renderings. The differences are highlighted with different colored boxes for different views (e.g. , the highlighted detailed structures of the Ship and Lego examples).
Figure 7
Ablation on the explicit mesh with different resolutions. By employing the explicit mesh as a guide for Gaussian learning, we evaluate the impact of varying mesh resolutions on the deformation results obtained from novel view points. We evaluate the performance of three different mesh resolutions: 50000, 100000, and 500000 vertices. The results indicate that our method is not sensitive to the mesh resolution when generating novel views of deformed 3D Gaussians.
Conclusions and Limitations
In this paper, we have proposed a novel large scale deformation method for 3D Gaussian Splatting, based on a mesh-based representation. The proposed 3DGS deformation method enables the manipulating of the 3DGS in an intuitive interactive manner.
To well facilitate the 3DGS deformation, we incorporate an explicit mesh that can be easily extracted by existing methods, which is bound with Gaussian ellipsoids together and enables the effective large-scale deformation of 3DGS.
In addition, we employ a Gaussian division model that operates on the explicit mesh through face split and normal guidance, which can improve the visual quality and prevents artifacts that may occur during large-scale deformations.
Based on the mesh-based GS representation, we introduce a large-scale mesh deformation method to enable deformable Gaussian Splatting by altering the parameters of the 3D Gaussians according to the user’s intuitive manipulations.
Despite this approach successfully achieving real-time largescale deformation with Gaussian Splatting representation, it still faces the following obstacles :
-
The visual appearance and shadow are still baked on the Gaussians and cannot be further edited.
-
This method relies on the extracted mesh as the proxy, it would fail if the mesh could not be extracted such as for complex transparent objects.
In future work, it is worth developing new methods that can not only deform the geometry of Gaussians, but also support editing of the appearance of Gaussians. In addition, our representation could also be applied to other applications such as digital human avatars, and further development for novel applications will be explored as future work.