Hierarchical Features Driven Residual Learning for Depth Map Super-Resolution 2019TIP 论文阅读
基于层次特征的残差学习在深度图超分辨率重建中的应用 2019 IEEE Transactions on Image Processing论文研读
Abstract
Abstract 随着负担得起的便携式消费式深度相机的迅速发展，在智能车辆和三维重建等许多计算机视觉任务中都得以利用深度信息。然而，由低成本深度传感器捕获的深度图（例如 Kinect）通常具有较低的空间分辨率，这限制了其潜在的应用。在本文中，我们提出了一种 novel 深度图超分辨率（SR）网络 DepthSR-Net。该网络能够自动从低分辨率（LR）版本推断出高分辨率（HR）深度图，并命名为 hierarchical features driven residual learning。具体而言，在残差 U-Net 深度网络架构的基础上构建了 DepthSR-Net。给定输入的低分辨率深度图后，在 首先通过三次样条插值上采样以获得所需的 HR 深度图 后构建输入金字塔以实现多级感受野效果。接着从输入金字塔、亮度图像以及 U-Net 的编码解码结构中提取层次特征。最后利用丰富的层次特征学习 HR 深度图与相应 interpolated 深度图之间的残差。将该残差加到插值后的深度图上即可获得最终 HR 深度图。我们通过一项消融研究展示了提出网络中每个组件的有效性。大量实验表明所提出的算法优于现有方法中的最先进算法。此外还探讨了该网络在网络视觉问题中的潜在应用前景
I. INTRODUCTION
II. RELATED WORK
A. Non Color-Guided Depth Map SR Method
B. Color-Guided Depth Map SR Method
C. Deep Learning-Based Color Image SR Method
With respect to existing studies, DepthSR-NET stands out as the most advanced method in terms of depth image super-resolution reconstruction.
III. PROPOSED METHOD
In this section, we initially succinctly define the problem that this study addresses. Subsequently, we comprehensively detail the architecture of our proposed DepthSR-Net model. Finally, we outline the loss function and provide an in-depth discussion of its training and implementation processes.
A. Problem Formulation
As per [44]'s conclusion, if the original mapping closely resembles an identity mapping, its optimization becomes significantly more straightforward. It becomes feasible to learn the residual between interpolated depth maps and their corresponding high-resolution depth maps by focusing on addressing omitted high-frequency details within bicubic interpolation upsampling processes.
B. Proposed DepthSR-Net Architecture
To illustrate the developed network topology and configuration parameters, Figure 2 provides a comprehensive overview.
• 输入金字塔支份数负责构建多层次感知域，并产出一套多层次表征;
• 编码器支份数整合自输入金字塔的多层次特征并产出一组层次编码体;
• 分级指导支份数提取分级强度特性和转化有用结构为最终HR深度图;
• 跳过连接传递至解码器回路中的编码体特性;
• 解码器支份数融合丰富层进式合成体形成剩余映射.
1) Input Pyramid Branch:
Input pyramid branch possesses notable advantages: (1) it enables the extraction of hierarchical feature representations derived from input depth maps; (2) it realizes multi-level spatial contextual information through multiple levels of receptive fields; (3) it minimizes the risk of over-fitting by representing features in an abstract manner.
2) Encoder Branch:
Unlike U-Net, our DepthSR-Net's encoder path merges hierarchical features extracted from input pyramid branches and fuses multi-level feature representations.
3) Hierarchical Y Guidance Branch:
不同于[4]中采用的多尺度制导方式，在本研究中我们采用了固定大小（即3 \times 3）的卷积核，并在解码器分支中指导残差图的重建过程。
通过使用3 \times 3卷积核提取分层强度特征，在重建过程中充分利用了强度图像中的不连续点来定位相关深度不连续点，并且有效降低了计算开销。
分层Y指导分支具有以下优势：
(1) 将强度图像中的层次结构信息转化为最终高分辨率深度图；
(2) 增加了解码器分支中的网络宽度。
4) Skip Connections:
Skip connection operation represents an additional link established between nodes across different neural network layers, effectively bypassing one or more intermediate nonlinear processing stages. The primary goal is to transfer corresponding features from the encoder branch to the decoder branch. The benefits of skip connections include: (1) effectively mitigating the vanishing gradient problem; (2) ensuring optimal information flow across layer boundaries; (3) promoting efficient reuse of learned features.
5) Decoder Branch:
In the decoder branch, we gradually integrate the decoder's features with those of other branches at a hierarchical level to generate a residual mapping.
C. Loss Function
