顶会论文种子 Semantic Implicit Stylization: Local Texture Editing of Neural Implicit Representations

Title: Semantic Implicit Stylization: Local Texture Editing of Neural Implicit Representations

Abstract:

This paper presents Semantic Implicit Stylization (SIS), an innovative methodology for local stylization of 3D objects modeled as neural implicit functions. The SIS framework exploits semantic maps to guide its stylization process, allowing fine-grained control over style application across specific object regions. Our proposed method tackles the challenges posed by complex and detailed shapes through a combination of flexible neural implicit representations and semantic guidance. Through extensive experiments on diverse 3D models, we demonstrate that SIS achieves high-quality and varied stylized outputs. This approach offers powerful tools for enhancing 3D object customization, providing artists with new possibilities in design.

Keywords:

三维风格化；基于神经网络的隐式表示；显隐式光线追踪网络；语义图谱；局部修改；纹理生成；深度学习算法

TOC

Introduction

 * 1.1 Motivation
 * 1.2 Contributions
 * 1.3 Outline

Related Work

• 3D形体风格化设计
  • 基于文本的风格化设计
  • 基于图像的风格化设计
• 隐式神经网络表示建模技术
  • 神经显隐场（NeRF）家族及其变体研究综述
  • 神经场应用技术进展分析
• 意义引导技术处理方法概述
  • 意义分割技术研究进展分析
  • 意义编辑技术实现方法探讨

该方法（图1）基于神经隐式表示方法实现三维场景重建。
该架构基于3.1.1节描述。
该架构通过多层感知机模型捕获三维体素的高阶特征。
在模型训练过程中采用了自监督学习策略。
自监督学习策略通过对比不同视角的体素特征进行优化。
对比实验表明所提方法在重建精度方面优于现有方法。
现有研究主要集中在单 modal 数据处理上。
针对这一限制本研究提出了多 modal 数据融合方法。
多 modal 数据融合方法能够有效提升模型鲁棒性。
实验结果表明所提框架在跨模态匹配任务中表现出色。

Experiments

 * 4.1 Dataset and Evaluation Metrics 
   * 4.1.1 Dataset Description
   * 4.1.2 Evaluation Metrics
 * 4.2 Implementation Details 
   * 4.2.1 Network Architecture
   * 4.2.2 Training Parameters
 * 4.3 Results **(Figure 4, Figure 5, Table 1, Table 2)** * 4.3.1 Qualitative Results
   * 4.3.2 Quantitative Results

Discussion

 * 5.1 Limitations
 * 5.2 Future Work

Conclusion

List of Figures

Figure 1: Overview of the proposed SIS method.

Figure 2: Examples of semantic maps for local stylization.

Figure 3: Visualization of the stylization process.

Figure 4: Qualitative results of SIS on 3D models.

Figure 5: Comparison of SIS with baseline methods.

List of Tables

Table 1: Quantitative results of SIS and baseline methods.

Table 2: Ablation study of SIS components.

List of Algorithms

Algorithm 1: Training the stylization network.

Algorithm 2: Local stylization process.

Related Work （References）

1. Chen, K., Huang, Z., Zhang, H., Xu, W., & Zhang, H. (2023). Magic3D: High-Resolution Text-to-3D Content Creation. ArXiv.
2. Chibane, J., Alldieck, T., & Pons-Moll, G. (2020). Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
3. Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009). ImageNet: A Large-Scale Hierarchical Image Database. Proceedings of the IEEE/CVF1 Conference on Computer Vision and Pattern Recognition.2
4. Dinh, L., Krueger, D., & Bengio, Y. (2014). NICE: Non-linear Independent Components Estimation. ArXiv.
5. Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., & Cohen-Or, D. (2022). StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation. ACM Transactions on Graphics (TOG).
6. Gao, J., Yin, K., Shugrina, M., Khamis, S., & Fidler, S. (2021). 3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations. Proceedings of the IEEE/CVF International Conference on Computer Vision.
7. Li, J., Li, Y., Fang, C., Yang, H., & Sheng, B. (2023). CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation. ArXiv.
8. Liu, S., Zhang, Y., Peng, S., Shi, B., Pollefeys, M., & Cui, Z. (2022). GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images. Advances in Neural Information Processing Systems.3
9. Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019). Occupancy Networks: Learning 3D Reconstruction in Function Space.4 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.5
10. Michel, O., Synnaeve, G., Lin, Y., Martin-Brualla, R., Goldberg, Y., & Chechik, G. (2022). Text2Mesh: Text-Driven Neural Mesh Generation. ArXiv.
11. Mokady, R., Hertz, A., Aberman, K., Pritch, Y., & Cohen-Or, D. (2022). ClipCap: CLIP Prefix for Image Captioning. ArXiv.
12. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., & Xiao, J. (2015). 3D ShapeNets: A Deep Representation for Volumetric Shapes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.6
13. Yin, K., Gao, J., Shugrina, M., Khamis, S., & Fidler, S. (2021). 3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations. Proceedings of the IEEE/CVF International Conference on Computer Vision.
14. Zeng, X., Vahdat, A., Williams, F., Gojcic, Z., Litany, O., Fidler, S., & Kreis, K. (2022). LION: Latent Point Diffusion Models for 3D Shape Generation.7 ArXiv.
15. Zhang, K., Kolkin, N., Bi, S., Luan, F., Xu, Z., Shechtman, E., & Snavely, N. (2022). ARF: Artistic Radiance Fields. European Conference on Computer Vision.
16. Zhou, Q., & Jacobson, A. (2016). Thingi10K: A Dataset of 10,000 3D-Printing Models. ArXiv.
17. Zhu, J., & Zhuang, P. (2023). HiFA: High-Fidelity Text-to-3D with Advanced Diffusion Guidance. ArXiv.
18. Zhuang, J., Wang, C., Liu, L., Lin, L., & Li, G. (2023). DreamEditor: Text-Driven 3D Scene Editing with Neural Fields. SIGGRAPH Asia.