StyleGaussian: Instant 3D Style Transfer with Gaussian Splatting(Related Work)
Radiance Fields
Radiance fields have recently made significant strides in advancing 3D scene representation. These fields are functions that assign radiance (color) and density values to arbitrary 3D coordinates. The __ color of a pixel__ is rendered by aggregating the radiances of 3D points through volume rendering.
Radiance fields find extensive applications across various domains of vision and graphics, especially in view synthesis, generative models, and surface reconstruction. They can be implemented through multiple approaches, such as MLPs, decomposed tensors, hash tables, and voxels , with numerous studies aiming to enhance their quality or rendering and reconstruction speed.
Among these advancements, 3DGS stands out for its rapid reconstruction capabilities, real-time rendering performance, and excellent reconstruction results.
① It models radiance fields using a multitude of explicitly parameterized 3D Gaussians.
② The cornerstone of its ability to render in realtime is its reliance onrasterization over ray tracing to render images.
Building on the strengths of 3DGS, StyleGaussian utilizes it to facilitate an immersive 3D editing experience.
3D Appearance Editing
Editing the appearance of traditional 3D representations like meshes or point clouds is generally straightforward, as meshes are associated with UV maps and points correspond to pixels in images.
However, editing radiance fields is challenging due to their implicit encoding within the parameters of neural networks or tensors.
Consequently, previous studies have resorted to learning-based methods for editing radiance fields, guided by images, text, or other forms of user input , encompassing modifications such as deformation, appearance changes, removal, relighting, and inpainting.
Yet, most of these approaches rely on a test-time optimization strategy , necessitating a time-intensive optimization process for each edit. Alternatively, some methods facilitate the editing of 3D scenes in a feed-forward manner. However, the editing speed of these approaches is still far from an interactive speed.
In contrast, StyleGaussian can edit the appearance of the scene instantly.
Neural Style Transfer
Neural style transfer aims to render a new image thatmerges the content structure of one image with the style patterns of another.
Prior research indicates that the second-order statistics of VGG features encapsulate the style information of 2D images. Initially, this field relied on optimization methods to align the style image’s VGG features, but subsequent approaches introduced feed-forward networks to approximate this optimization process, significantly enhancing the speed of style transfer.
More recent efforts have extended style transfer to the 3D domain by attempting to stylize point clouds or meshes. However, these methods often lag in rendering capabilities when compared to radiance fields, prompting further research into the stylization of radiance fields.
Works have managed to achieve radiance field style transfer through optimization, offering visually impressive stylizations but at the cost of time-consuming optimization for each new style and limited generalizability to unseen styles.
Alternatives such as HyperNet embed style information into MLP parameters for arbitrary style transfer but face slow rendering and poor detail in style patterns.
StyleRF introduces zero-shot radiance field style transfer but uses a 2D CNN decoder , impairing multi-view consistency.
However, StyleGaussian allows for instant transfer and real-time rendering while maintaining strict multi-view consistency.
