Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting

Abstract

The recent advancements in 3DGS have not only facilitated real-time rendering through modern GPU rasterization pipelines but have also attained SOTA rendering quality. Nevertheless, despite its exceptional rendering quality and performance on standard datasets, 3D-GS frequently encounters difficulties in accurately modeling specular and anisotropic components. This issue stems from the limited ability of spherical harmonics (SH) to representhigh-frequency information.

To overcome this challenge, we introduce Spec-Gaussian, an approach that utilizes an anisotropic spherical Gaussian (ASG) appearance field instead of SH for modeling the view-dependent appearance of each 3D Gaussian.

Additionally, we have developed a coarse-to-fine training strategy to improve learning efficiency and eliminate floaters caused by overfitting in real-world scenes.

Our experimental results demonstrate that our method surpasses existing approaches in terms of rendering quality. Thanks to ASG, we have significantly improved the ability of 3DGS to model scenes with specular and anisotropic components without increasing the number of 3D Gaussians.

Figure

Figure 1

Our method not only achieves real-time rendering but also significantly enhances the capability of 3DGS to model scenes with specular and anisotropic components.

Key to this enhanced performance is our use of ASG appearance field to model the appearance of each 3D Gaussian , which results in substantial improvements in rendering quality for both complex and general scenes.

Moreover, we employ anchor Gaussians to constrain the geometry of point-based representations , thereby improving the ability of 3DGS to accurately model reflective parts and accelerating both training and rendering processes.

Figure 2

Pipeline of our proposed Spec-Gaussian.

The optimization process begins with SfM points derived from COLMAP or generated randomly, serving as the initial state for the anchor Gaussians.

Within a view frustum, neural Gaussians are spawned from each visible anchor Gaussian , using the corresponding offsets.

Their other attributes(such as opacity , rotation , and scaling ) are decoded through the respective tiny MLPs.

To address thelimitations oflow-order SH and pure MLP in modeling high-frequency information , we additionally employ ASG in conjunction witha feature decoupling MLP to model the view-dependent appearance of each neural Gaussian.

Subsequently, neural Gaussians with opacity are rendered through a differentiable Gaussian rasterization pipeline , effectively capturing specular highlights and anisotropy in the scene.

Figure 3

Using a coarse-to-fine strategy , our approach is able to optimize the scene in a progressive manner and eliminate the floaters efficiently.

Figure 4

Visualization on NeRF dataset.

Our method has successfully achieved local specular highlights modeling , a capability that other 3DGS-based methods fail to accomplish, while maintaining fast rendering speed.

Compared to Tri-MipRF, a NeRF-based method , we have significantly enhanced the ability to model anisotropic materials.

Figure 5

Ablation on ASG feature decoupling MLP.

We show thatdirectly using ASG to model color leads to the failure in modeling anisotropy and specular highlights.

By decoupling the ASG features through MLP , we can realistically model complex optical phenomena.

Figure 6

Visualization on Mip-NeRF 360 dataset.

This clearly demonstrates that our method is capable of modeling complex specular highlights and effectively removing floaters , outperforming other methods in these aspects.

Figure 7

Visualization on our anisotropic dataset.

We have demonstrated the superiority of our method compared to 3DGS and scaffold-GS, which models color based on MLP.

With the help of ASG, we can model specular highlights and anisotropic parts of the scene more effectively.

Figure 8

Visualization on NSVF dataset.

Our method significantly improves the ability to modelmetallic materials compared to other GS-based methods. At the same time, our method also demonstrates the capability to model refractive parts, reflecting the powerful fitting ability of our method.

Figure 9

Ablation on anchor Gaussians.

The shiny scene is borrowed from Nex. This clearly demonstrates that anchor Gaussians can improve the geometry of 3DGS. Consequently, this enhancement aids in its ability to learn the reflective parts of the scene, as highlighted in the orange and blue boxes.

Figure 10

Ablation on coarse-to-fine training mechanism.

Experimental results demonstrate that our simple yet effective training mechanism can effectively remove floaters in both the background and foreground, thereby alleviating the overfitting problem prevalent in 3DGS-based methods.

Conclusion

In this work, we introduce Spec-Gaussian, a novel approach to 3DGS that features an anisotropic view-dependent appearance.

Leveraging the powerful capabilities of ASG, our method effectively overcomes the challenges encountered by 3DGS in rendering scenes with specular highlights and anisotropy.

Additionally, we innovatively implement a coarse-to-fine training mechanism to eliminate floaters in real-world scenes.

Both quantitative and qualitative experiments demonstrate that our method not only equips 3DGS with the ability to model specular highlights and anisotropy but also enhances the overall rendering quality of 3DGS in general scenes, without significantly compromising FPS and storage overhead.

Limitations

Although our method enables 3DGS to model complex specular and anisotropic features, it still faces challenges in handling reflections.

Specular and anisotropic effects are primarily influenced by material properties , whereas reflections are closely related to the environment and geometry.

Due to the lack of explicit geometry in 3DGS, we cannot differentiate between reflections and material textures using constraints like normals, as employed in Ref-NeRF and NeRO.

In our experiments, we also observed that whenground truth geometric information is provided , 3DGS becomes more consistent with expectations under strict constraints , but this comes at the cost of a certain decline in rendering quality.

We plan to explore solutions for modeling reflections with 3DGS in future work.