Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering
Abstract
We propose Gaussian Frosting, an innovative grid-based representation for high-quality rendering and editing of complex 3D effects in real-time. Our approach builds on the recent 3DGS, which optimizes a set of 3D Gaussians to approximate the radiance field from images.
Firstly, we suggest extracting abase mesh from Gaussians during the optimization process. Subsequently, we construct an adaptive layer of Gaussians with variable thickness around this base mesh. The refined layer effectively captures fine details and volumetric effects, such as those seen in hair or grass.
We refer to this layer as Gaussian Frosting, similar to how frosting adorns cakes. As the material's softness increases, so does the thickness of this frosty layer.
We also present a parameterization of the Gaussians to ensure they remain within the frosting layer, automatically modifying their parameters during deformation, scaling, modification, or animation processes.
This representation enables efficient rendering through GS. Additionally, it supports performing manipulation and generating animations via altering the base mesh.
We evaluate the performance of our method across diverse synthetic and real-world scenarios, demonstrating its effectiveness. We also showcase that our approach significantly outperforms existing surface-based techniques in terms of accuracy and efficiency.
Figure
Figure 1
We propose to describe surfaces using a mesh surrounded by a frosting layer of variable thickness, made up of 3D Gaussians.
This representation effectively captures intricate volumetric phenomena created by soft materials such as cat's hair, grass, and smooth surfaces. The system relies solely on RGB imagery, rendering in real-time and enabling animation through conventional techniques.
In the example above, we successfully animated both Buzz and the kitten, altering their original poses**(a)** while maintaining high-fidelity rendering (b). In contrast to SuGaR, the highly detailed fur of the kitten covers Buzz's legs in a lifelike manner**(c)**.
Figure 2
Visualization of the thickness of theFrosting layer on an example.
More intense layers are characterized by higher brightness values. Our method automatically constructs a Frosting layer for soft regions, such as (e.g., the fur of a red panda plush), and a thin Frosting layer for flat surfaces, such as tables or floors.
Adjusting the thickness of the frosting layer permits distributing more Gaussians in regions where increased volumetric rendering is required near the surface, thereby achieving an efficient distribution of Gaussians across the scene.
采用可变厚度能带来更高的性能；不仅在动画过程中减少了 artifact 的出现。
Figure 3
Scene composition.
By applying mesh editing techniques within the Blender suite, we could seamlessly integrate various elements from within different scene layers (a) to construct an entirely new scene (c).
We also changed the pose of the characters by using the rigging tool in Blender**(b)**.
Similarly to surface-based approaches such as SuGaR, Frosting offers versatile tools for scene editing and compositing, enabling more effective rendering of complex volumetric effects and fuzzy materials like hair or grass.
Figure 4
Creating a Layer of Gaussian Frosting.
We begin by performing precise optimization on a Gaussian Splatting representation, employing a rendering loss, while imposing no extra constraints, to enable the Gaussians to self-position effectively. We refer to these unconstrained Gaussians as its Gaussian nature allows for...
After adjusting these Gaussians to ensure their alignment with the surface, we generate or create a mesh that will function as the foundation for the frosting.
Following this approach, we utilize the misalignment of surface-aligned Gaussians as a key indicator to determine regions requiring additional volumetric rendering techniques while establishing interval ranges for further analysis.
around the mesh’s vertices
.
Finally, it is achieved through applying the density function of unconstrained Gaussians to adjust and narrow down the intervals, resulting in a Frosting layer.
As a result, we sample a novel, densified set of Gaussians inside the layer.
Figure 5
Comparative analysis of meshes extracted by SuGaR from the Shelly dataset, which provides a clear distinction between whether or not our improvement is applied, in Poisson reconstruction where octree depth D is automatically optimized based on scene complexity.
改写说明
Figure 6
How we define the inner and outer bounds of the Frosting layer.
Figure 7
Rendering complex scenes with Frosting.
(a) Renderings.
(b) recovered normal maps.
(c) estimated Frosting thickness.
Note that the Frosting isstiff on soft materials (among examples of which are hairs and grasses), and light on smooth surfaces (such as tables).
Figure 8
Close-up perspectives of vaguely textured substances were reconstructed from the Shelly dataset.
Figure 9
Examples of animation with Frosting.
We successfully brought the sculpture to life within the left-hand image by utilizing its rigging tool through the software platform named Blender.
Figure 10
Comparison with a constant thickness.
Adopted method for computing Frosting's adaptive thickness is crucial to ensuring optimal performance and eliminating potential artifacts during scene editing.
(c)
Conclusion
We introduced an elegant and robust surface representation that offers numerous benefits compared to existing representations, along with a method for extracting it from images.
One weakness of our approach is the elementary deformation model, which relies on a piecewise linear approximation. It is easy to replace it with a more accurate and physically based deformation model.
Another drawback is that our models exceed in size those of 3DGS due to the necessity to incorporate barycentric coordinates and mesh vertices. Recent studies on techniques to reduce the size of 3DGS have shown promising results.
Computer Graphics researchers普遍认为, Frosting表示法可能超越基于图像渲染的应用,并展现出其潜力.它甚至可以在更为广泛的计算机图形学领域中得到应用,用于实现复杂材料的实时渲染.