毫米波成像 论文阅读笔记 | HawkEye, CVPR 2020

原文链接:https://mp.weixin.qq.com/s/KyMyyZert7ZYltfZaYrBWA

论文阅读笔记：其中包含作者Jianfeng Guan及其合著者Siddharth Madani、Sachin Jog、Sumeet Gupta以及Hassan Hassanieh的研究成果，在2020年CVPR会议上获得了广泛关注。通过毫米波雷达在低能见度环境下实现高分辨率成像的研究成果，在2020年CVPR会议上获得了广泛关注

Abstract

• mmWave high-resolution imaging: in dense fog
• Motivation *

mmWave signals:

The system exhibits good propagation properties under low visibility circumstances.

❌ suffer from very low resolution, specularity, and noise artifacts.

光学传感器则反之

• 本文采用HawkEye技术基于生成对抗网络（Generative Adversarial Networks, GANs）实现对低分辨率毫米波热图形状信息的恢复。
• 该系统基于毫米波信号的结构特征及其本性进行设计。
• 开发了一个数据合成器：构建训练数据集。
• 将我们的系统部署于自研的毫米波雷达平台并展示了性能提升。

1 Introduction

mmWave Radar high-resolution imaging的意义

• Significance lies in severe weather conditions (dense fog, smog, snowstorms, and sandstorms) and low-light scenarios (poornight vision scenarios) necessitating the use of millimeter-wave radar for autonomous driving.
• several challenges lie in however, millimeter-wave radar imaging remains a significant challenge (as shown in figures d and e)
• 1 Lower resolution characteristics pose a problem.
• 2 Highly specular reflections complicate the imaging process.
• 3 Multiple path propagation issues lead to shadow reflections and artifacts across various regions.

• 现有的解决方案：
• 1 提高分辨率：采用人尺寸机械可调节阵列。
• 2 消除多径反射：将被成像对象隔离在近场区域。
• 3 解决镜面效果：旋转阵列以围绕被成像对象。
• 显然，在自动驾驶应用中这一设计方案显得过于笨重且不切实际。

本文工作 HawkEye 概述

Key idea :

frame the challenge of predicting high-frequency shapes from raw mmWave heatmaps as a learning task.

Advantages :

Use of learning provides robustness *

efficiently utilize the priors related to the shapes of cars to generate plausible forecasts

Innovations *

the design of the Neural Network : map 3D input heatmaps to 2D depth maps

Loss function : combination of perceptual, L1, and adversarial loss

Training Data :

A realistic radar data generator capable of reflecting distinct features of radar systems.

A real-world dataset aggregation system is designed to gather accurate and up-to-date datasets for refining machine learning models and establishing performance benchmarks.

System Overview (四个模块,如下图)

Module 1 : handcrafted mmWave imaging module \Rightarrow acquire radar data stream

Module 2: A stereo camera system with a wide baseline, which \Rightarrow results in the acquisition of the ground truth high-resolution 2D depth map.

课程3 : 合成器 ⇒ 基于汽车CAD模型和毫米波射线追踪算法生成合成数据

Module组别 : 生成对抗网络(GAN) ⇒ 生成高分辨率深度图并从原始的3D毫米波热图中重建汽车在真实场景中的雾景

2 Related Work

2.1 Super-Resolution

该技术基于image patch correspondences between low-resolution and high-resolution image patches.

点云超分辨 (closest to this work)

* upsamplng sparse 3D LiDAR data to create dense 2D depth maps

上述工作的特点(优势) ：

• 或 同时利用cameras与LiDAR进行操作
• 或 主要依赖于高频视觉特征（如边缘）来分类物体并增强细节

毫米波超分辨率在实现高分辨率方面面临诸多挑战, 其主要原因在于显著降低的空间分辨率。尽管如此, 高频视觉特征如边界和边缘等细节信息在毫米波频段中不易察觉。此外, 常规采样与超分辨率技术难以有效消除由这些现象带来的干扰

2.2 LiDAR in Fog

现有工作的limitations

• must know the depth information about the scene in advance
• involve estimating, from a stationary object, the statistical distribution of photons it reflects
• only capable of detecting with limited depth, resolution, and field-of-view

但毫米波没有这些限制

2.3 Radar Imaging Systems

现有 high resolution radar imaging system 的主要缺点 是其应用范围受限（\Rightarrow 仅在非常近的距离范围内（<50厘米））。系统中的 bulky scanners 造成了较大的体积占用，并通过 The system integrates advanced optical components 实现了高分辨率成像功能。然而，在移动平台上运行时表现出显著的局限性

现有基于深度学习的工作 * 分辨率仍较低
* 短距离工作

3 Millimeter Wave Imaging Background

• 3D热图的生成 * 4个 Fast Fourier Transform (FFT)

• 3D heatmap: x(φ, θ, ρ) \Rightarrow 其中 x(φ, θ, ρ) 表示空间中每个体素反射的能量分布

  • An example:

• Challenges of mmWave radar imaging (compared to camera and LiDAR)

1 Low imaging resolution is (significantly lower than vision)

❌ range resolution (10cm, 3.3* worse than LiDAR)

❌ 水平角和垂直角分辨率（分别为5度和50分）比LiDAR低 \Rightarrow 需要安装至少9米长的天线阵列 才能获得与LiDAR相似的亚度数分辨率

2 Specular reflections

✅ Most reflections never trace back to the mmWave receiver

❌ 导致 Sparsity + Multipath artifacts

The radar data exhibits distinct representations and perspectives compared to cameras.

设计的网络须 accommodate the above challenges

4. HawkEye’s Architecture

4.1 Overview

based on conditional GAN

Generator *

input: mmWave heatmap x

🚩 Low spatial resoltion, but high depth resolution

encoder-decoder architecture

skip-connections: retain highfrequency details in depth

Discriminator

Loss * GAN loss
* L_1 + perceptual loss

在设计过程中 考虑到了 peculiarities of the raw mmWave signal 的特性

4.2 Input and Output Representation

• Input :

  • 3D radar heatmap (φ, θ, ρ) \in \mathbb{R}^{64\times 32 \times 96}

• Output :

  • 2D depth map of the target \in \mathbb{R}^{256\times 128}

4.3 Generator Architecture

• encoder

  • latent vector length: 2048
  • 3D Conv + Leaky-ReLU + BN

• decoder

  • Deconv + Leaky-ReLU + BN
  • tanh + FC

4.4 Skip Connections

• Motivation * Keep the high-frequency information from the input

• 做法 *

step 1: projects the input 3D heatmap into a 2D image

Recording the location corresponding to the highest value: （选择前八项）, each channel consists of eight subchannels.

x_{2 D}(\phi, \theta)=\arg \max _r x_{3 D}(\phi, \theta, r)

step 2: concatenated with the features maps at the 6th layer in the decoder

Such a non-differentiable operation is exclusively performed with the input.

4.5 Discriminator Architecture

• 如下图

4.6 Loss Function

生成器损失由三部分构成：其中包含传统GAN的损失函数\mathcal{L}(G)；基于L_1范数的损失项（衡量真实值与预测值之间的差距）；以及感知损失项\mathcal{L}_p(G)。值得注意的是，在讨论生成器损失时

\downarrow

\begin{aligned} &\hat{\ell}_{1}(G) 实现为 \ell(y-G(x)) \\ &\hat{\ell}_{p}(G) 实现为 \ell(VGG(y)-VGGG(x)) \\ &\hat{\ell}_{H}(G) 实现为 \ell(G) + \lambda_{1} \ell_{1} + \lambda_{p} \ell_{2} \end{aligned}

5. HawkEye’s Data Synthesizer

• Input :

  • 3D CAD 模型

• Output :

  • (3D mmWave heatmap + 2D depth) pairs

• 特点：该系统旨在创建汽车的三维点反射模型

• 使用射线追踪方法模拟毫米波雷达信号

• 考虑了多路径反射以及基于反射角度的镜面效应

• 生成效果显著优异

5.1 step 1: Scene Generation

• based on two types of datasets :

1 3D CAD models

✅ precise 3D meshes of a wide variety of cars

2 Cityscapes, a street view video recordings dataset

✅ offer references for car placement in the camera frame

✅ 利用Mask R-CNN 检测目标

5.2 step 2: Ray Tracing

model the mmWave reflectors in the scene

• represent the remaining parts as clusters of point reflectors
  • construct standard ray tracing [16] on the point reflectors

5.3 step 3: mmWave Heatmap and Ground-truth Generation

• 基于点反射器模型
• 引入背景噪声
• 非平凡的外校准
• 获取（三维毫米波热图与二维深度图）配对

6. Experiments

6.1 Dataset and Implementation

Dataset

Radar data

• Simulate the behavior of a two-dimensional (2D) antenna array using a 60 GHz radio.
• Utilize conventional radar waveform techniques to generate three-dimensional (3D) millimeter-wave (mmWave) heatmaps.

2D depth map

• Calibrated iPhone Camera
• 利用标准的立体图像处理技术 [35] 生成二维深度图
• 随后通过Mask-RCNN去除不属于车辆的 voxels

真实采集数据： 包含327个样本
包含三种不同的场景：室内停车场、露天停车场以及露天车辆通道
分为九个类别和六十分类车辆：包括亚 compacts（2辆）、compacts（12辆）、mid-sized（16辆）、full-sized（7辆）、sports cars（5辆）、SUVs（11辆）、一辆JEEP、两辆Vans和四辆Trucks

合成数据集 * 4000 samples on 120个car models

Controlled Experiments in Fog/Rain

• 101/327 个真实数据 from fog and rain
  • 雾和雨的产生：
    • 使用高密度水基烟雾设备配合的烟雾生成器
      • 通过在汽车周围有限区域使用的水 hoses进行模仿

Training

• 模型在训练过程中分为两个阶段
• 第一个阶段：利用大量 synthetic 数据进行学习
• 第二个阶段：利用真实数据进行微调以提升模型性能。

Baselines

• 1 原始毫米波热图
• 基于 L_1 网络（用于分析GAN损失）
• Closest : 在输入特征空间中检索3D雷达热图中的样本点，并采用最小欧氏距离实现检索。
  • 该网络并非单纯地存储数据特征

6.2 Qualitative Results

Failure Examples

• 该系统难以处理强虚反射和表象现象在热图中的表现
  • 难以处理复杂的多车交通场景
    • 当场景包含多个车辆时 效果会下降

6.3 Quantitative Results

7 Conclusion

虽然HawkEye仍有改进空间，在二维深度图重构方面取得了显著进展