文献速递：深度学习肝脏肿瘤诊断---动态对比增强 MRI 上的自动肝脏肿瘤分割使用 4D 信息：基于 3D 卷积和卷积 LSTM 的深度学习模型

Title

题目

Automated Liver Tumor Segmentation on Dynamic Contrast Enhanced MRI Incorporating Four-Dimensional Imaging Data: With the Aid of a Deep Learning Model, Particularly Employing Three-Dimensional Convolutions and a Combination of Convolutional LSTM Networks.

动态对比增强技术在MRI图像中实现肝脏肿瘤的自动分割，并通过引入四维数据来提升分割效果；采用三维卷积层与卷积长短期记忆网络相结合的深度学习架构进行建模。

01

文献速递介绍

作为最常见的癌症类型之一，肝癌在癌症相关死亡中占据显著地位。HCC即为最常见的原发性肝癌类型，在全球恶性肿瘤排名中位列第五，在癌症相关死亡原因中位居第三。精准的肿瘤切除手术对早期HCC诊断及治疗具有重要意义。通过精确分割技术获取基于体积的定量指标有助于提升诊断准确性，并为治疗方案制定提供科学依据。这些定量分析包括纹理特征等指标，在肝脏肿瘤学研究中扮演着重要角色，并能辅助医生进行更精准的治疗反应评估、肿瘤分期以及患者预后分析。

目前，在肝癌诊断中仍大量采用人工轮廓绘制的方法来进行肝脏肿瘤分割工作。这种方法虽然直观但具有较大的工作量且容易受不同操作者间的技术差异及同一操作者多次操作的一致性问题影响。为此已提出多种基于传统图像处理算法的计算机辅助诊断手段包括但不限于以下几种技术：基于阈值的方法、基于空间正则化的技术、监督学习分类方法以及无监督聚类分析技术等用于实现肝癌病变区域的空间定位与形态描述。然而，在实际应用中遇到的主要挑战包括肿瘤形态及外观特征的显著变化性边界不明确以及引入对比剂所带来的额外噪声干扰等问题

前述方法的主要局限性在于其仅依赖于有限的参数。如仅强度信息这一单一指标，在模糊肿瘤边缘处出现漏判现象。

近年来，在医学图像分析领域中应用了深度学习技术，并取得了显著进展。例如，在脑肿瘤分割和前列腺癌检测等方面已经取得了突破性成果。此外，在肝脏成像相关的任务中也广泛采用了深度学习技术来解决健康肝脏分割、肝纤维化分期、肝脂肪浸润分类以及肝脏肿瘤诊断等难题。作为一种基于数据驱动的方法，深度学习能够从大量图像数据中提取关键特征并提升相关任务的表现水平。同时，在肝脏肿瘤分割这一领域上也取得了令人瞩目的成就

Abstract

摘要

Objective: High-quality identification of liver tumors plays a pivotal role in guiding clinicians in selecting suitable therapeutic strategies and evaluating the efficacy of surgical interventions. This study introduces a novel four-dimensional (4D) deep learning framework, integrating three-dimensional convolutional neural networks with convolutional long short-term memory (C-LSTM), specifically tailored for the segmentation of hepatocellular carcinoma (HCC) lesions. Methods: This innovative approach leverages the comprehensive temporal and spatial features embedded within dynamic contrast-enhanced magnetic resonance imaging (MRI) data. A dedicated three-dimensional CNN module extracts salient features across individual time phases, followed by a four-layer C-LSTM network that captures temporal dependencies. By synergistically combining multi-phase MRI information with tissue feature evolution across diverse imaging modalities, our model achieves robust performance in lesion characterization. Results: Our method demonstrated exceptional performance, attaining a Dice score of 0.891 ±0.080 for volumetric accuracy in liver tumor segmentation tasks. These metrics significantly outperform those achieved by conventional approaches such as traditional three-dimensional U-nets and recurrent U-nets, while maintaining computational efficiency through optimized training protocols. Conclusion: The proposed framework represents a significant advancement in automated tumor segmentation techniques, offering reliable solutions for clinical practice while maintaining computational efficiency. Index Terms: Four-dimensional information; deep learning; three-dimensional convolution; convolutional LSTM; tumor segmentation

核心任务

Conclusions

结论

I n this paper, we developed a 4D deep learning model

for automatic liver tumor segmentation, which offers better performance than some other networks. Our method leverages a 3D CNN module to extract 3D spatial context and a C-LSTM network to process temporal information, thereby integrating 4D data to aid segmentation. Experimental results demonstrate that our proposed model achieves improved performance in ablation experiments compared to existing state-of-the-art models while significantly reducing prediction time. The accurate segmentation of liver tumors serves as an essential prerequisite for subsequent quantitative analysis and provides substantial benefits for clinical diagnosis and treatment.

在本研究中

Method

方法

A. Dataset and MRI Protocol This study was approved by the research-ethics commit tee of First Affiliated Hospital of Zhejiang University. The clinical, radiological, and histopathological data were col lected from medical charts. The retrospective study included 190 pathologically confirmed primary HCC patients who underwent liver MRI scanning before surgery between January 2017 and March 2020. A fat-suppressed 3D T1-weighted GRE sequence was performed on a 3.0 T clinical scanner (GE Signa HDx; GE Healthcare). Gadopentetate dimeglumine (Magnevist; Bayer Healthcare, Germany, 0.1 mmol/kg) was injected at a rate of 2.5 ml/s followed by saline flush with a maximum dose of 18 mL. Images in the hepatic arterial, portal venous, and delayed phases were obtained at 25∼35 s, 55∼75 s, and 180∼240 s after contrast medium injection respectively. The scanning parameters are as follows, echo time (TE): 1.5 ms; repetition time (TR): 3.2 ms; In-plane resolution: 0.8 × 0.8 mm2; slice thickness: 2.5 mm; matrix size: 320 × 256; number of slices: 84; and field of view (FOV): 400 × 400 × 210 mm3. This data was randomly split into a training set (110 cases), a validation set (40 cases) and an internal test set (40 cases). In addition, we further included 60 HCC DCE data from Fudan University Affiliated Zhongshan Hospital as an external test set for this study. MR scanning was performed on a 3.0 T Siemens scanner (Magnetom Verio 3T MRI, Siemens Healthineers) with 3D gradient-echo VIBE sequence. The contrast, injection rate, and image acquisition timepoints were the same as for the internal dataset. The scanning parameters are as follows, TE: 1.4 ms; TR: 4.1 ms; In-plane resolution: 1.1 × 1.1 mm2; slice thickness: 3.0 mm; matrix size: 352× 260; number of slices: 72; and FOV: 269× 380 ×180 mm3. The target livers and HCC lesions were outlined by an expe rienced radiologist (with 15 years of experience in abdominal imaging) using ITK-SNAP (v3.6.0) in the delayed phas, with reference to the pre-contrast, arterial and portal venous phases. In addition, another radiologist (with 15 years of experience in abdominal imaging) checked and adjusted the outlined labels, and if there was no agreement on a particular area, a thirdradiologist (with 30 years of experience in liver imaging) would make the final decision.

A. 数据集和 MRI 协议

本研究经浙江大学第一附属医院研究伦理委员会审核。

此外，在本研究中，我们进一步增加了来自复旦大学附属中山医院的60例HCC DCE数据作为外部测试集。详细地说，在本研究中采用的扫描参数包括：TE值设定为1.4 milliseconds；TR值设定为4.1 milliseconds；平面分辨率设置在每个方向上均为1.1 × 1.1 mm²；层厚度设定为3 millimeters；矩阵尺寸定位于352 × 260 pixels；切片数量设置为72 slices；整个扫描区域的空间分辨率为269 × 380 × 180 mm³。

目标肝脏和 HCC 病变的描绘由一名经验丰富的放射科专家（拥有15年腹部影像学专业经验）使用 ITK-SNAP (v3.6.0) 在经检查阶段绘制，并参考了无对比剂期、动脉期及门静脉期的影像数据。此外，在另一名同样具备15年腹部影像学经验的放射科专家对图像标记进行优化调整时（若对某些区域存在分歧），第三位拥有丰富肝脏影像学经验的放射科专家将负责做出最终决策。

Figure

图

Fig. 1 illustrates the overall framework of the proposed 4D deep learning model for HCC segmentation, comprising a 3D CNN module and a C-LSTM network module. Simple 3D U-net modules were utilized in the pre-contrast, arterial, portal venous, and delayed phases to separately extract spatial domain information. A four-layer Conv-LSTM network was designed to exploit time domain information across multiple DCE phases. In the Conv-LSTM network block, m denotes the number of C-LSTM network layers and is equal to 4.

图1展示了所提出的用于HCC分割的四维深度学习模型的整体架构。该架构包含两个关键组件：一个是三维卷积神经网络（CNN）模块（以粉色表示），另一个是循环时序神经网络（LSTM）模块（以绿色表示）。通过浅层三维U-Net架构，在四个不同的时间点——无对照期、动脉期、肝静脉期以及延迟期——提取了空间信息。其中，在Conv-LSTM网络中使用的m代表该LSTM网络的层数级数，在此实现中设定为4层。

_Fig. 2. 4D information in the proposed deep learning model: 七张连续切片的图像中提取出的三维空间上下文特征（通过三维卷积完成），同时结合了基于四相位DCE图像的时间域信息（通过C-LSTM提取）。总共从28张图像中提取的信息用于预测目标切片处的肿瘤掩膜。

图2展示了该深度学习模型处理的4D信息。每个DCE阶段经过七张连续切片图像获取了三维空间信息（利用三维卷积）。并经四阶段DCE过程获取时间维度数据（借助C-LSTM）。整体整合了28张切片数据用于预测目标区域肿瘤掩膜。

Fig. 3. Network architecture of the 3D U-net based liver segmentationmodel.

图 3. 基于 3D U-net 的肝脏分割模型的网络架构。

Fig. 4. Training and testing strategies. (a) Liver segmentation, training:image patches of size 16 × 256 ×256, with 8 slices of overlap; testing:image patches of size 16 ×256 × 256, with 8 slices of overlap, retaining the prediction results of middle 8 slices;(b) tumor segmentation, training: image patches of size 7 × 224 × 256, generating training candidates z for all tumor slices, with one out of every three for non-tumor slices, and cropping seven consecutive slices centered on each candidate (from z−3 to z+3); testing: image patches of size 7 × 224 × 256, generating testing candidates for all image slices, and predicting theintermediate slice for each image patch.

图 4. 训练与验证策略。(a) 肝脏分割：采用16×256×256像素大小的图像块进行训练与测试操作；每次分割时会在相同位置产生8个连续切片用于验证；(b) 肿瘤分割：在训练阶段采用7×224×256像素大小并生成训练候选区z；从所有肿瘤切片中选取具有代表性的样本作为候选区域；在测试阶段则为每个图像生成相应的候选区域，并对每个切割出的7×224×256像素块进行中心位置预测

_Fig. 5展示了网络输入与基本模块的不同组合方式,(a)在所提出的模型中为每个DCE阶段分配独立的基本模块,Basic+LSTM模型;(b)具有共享权重的基本模块,Basicshare+LSTM模型;(c)具有多通道输入的基本模块,Basicstack+LSTM模型.I代表图像,F代表特征图,DYN1至DYN4分别对应预对比、动脉、portal静脉和延迟相位.]

图5展示了网络输入与基础模块的不同组合模式。(a)中我们构建了该模型中每个DCE阶段的独立基础模块，并采用Basic+C-LSTM架构；(b)中则采用了共享参数权重的基本模块，并基于Basicshare+C-LSTM框架展开；(c)中引入多通道输入机制，并以Basicstack+C-LSTM模式实现。其中I代表图像数据,F表示提取的特征图,DYN1-DYN4分别对应无对比剂时期、动脉期、门静脉期以及延迟期的时间序列特征。

The ablation studies for the proposed model are illustrated in Figure 6, comprising four distinct configurations. The first category involves models employing single-phase DCE inputs, specifically BasicDYN1 through BasicDYN4. The second category represents the baseline model that does not incorporate the C-LSTM structure. The third and fourth categories introduce enhanced versions featuring information interaction truncation: one integrating early fusion (Basic+CNNEF) and the other implementing late fusion (Basic+CNNLF). Notably, the original C-LSTM module is substituted by a 2D CNN architecture in these experiments.

本研究中的消融实验采用了四个不同模型架构进行对比分析。(a) 首先是基于单阶段 DCE 的模型架构设计；(b) 本研究还引入了不含C-LSTM结构的网络模块进行性能评估；(c) 通过截断信息交互机制采用C-LSTM结构，并结合早间特征融合的方式优化了模型性能；(d) 进一步地，在(c)的基础上改变了信息融合的时间点，在晚间特征融合的情况下取得了更好的实验结果。其中虚线框区域代表原始C-LSTM网络模块，在本研究中被二维卷积（2D CNN）结构取代。

Fig. 7展示了基于所提模型和其他模型在消融实验中的肝肿瘤分割结果。(a)(b)来自内测集的案例;(c)(d)来自外测集的案例。从左至右依次展示了预对比相、动脉相、门静脉相及延迟相的分割结果。绿色轮廓：人工标注的掩膜(参考标准);蓝色掩膜：基于所提模型预测的结果;黄色掩膜：基于基本模型预测的结果;红色掩膜：基于基本+CNNLF改进模型预测的结果.

图 7. 根据所提出的模型及其消融实验中其他模型的肝脏肿瘤分割结果展示如下：(a) 和 (b) 代表内部测试集的 Pat #1 和 Pat #2 案例；(c) 和 (d) 代表外部测试集的 Pat #3 和 Pat #4 案例。从左到右依次为无对比剂阶段、动脉期、门静脉期和延迟期的分割结果展示。其中绿色轮廓标注为人工标记的标准（真实情况），蓝色区域为所提出模型预测的结果边界框，黄色区域为基本模型预测的结果边界框，红色区域则由基本+CNNLF 模型预测得到的结果边界框所限定。

Fig. 8. Liver tumor segmentation outcomes compared to the proposed model and some external baseline models. (a) Pat #1 and (b) Pat #2 cases derived from the internal validation set; (c) Pat #3 and (d) Pat #4 cases derived from the external validation set. From left to right: segmentation outcomes demonstrated in pre-contrast phase, arterial phase, portal venous phase and delayed phase. Contour displayed in green represents manually annotated masks (ground truth), while blue indicates masks predicted by the proposed model, yellow denotes masks predicted by nnU-net model, and red signifies masks predicted by RA-Unet model. A comparison with the 3D U-net was omitted since it exhibited significantly inferior performance.

图 8. 基于所提出的模型以及若干外部基准模型对肝脏肿瘤进行分割实验的结果展示。(a) 来自内部测试集的病例 1 和 (b) 病例 2 的对比；(c) 来自外部测试集的病例 3 和 (d) 病例 4 的对比。实验结果涵盖无染色阶段、动脉期、门静脉期及延迟期的表现。其中绿色轮廓代表人工标注的真实情况（参考标准），而蓝色区域则表示所提出模型预测的结果；黄区为 nnU-net 模型预测区域；红区为 RA-Unet 模型预测区域。（与 3D U-net 的比较未展示因其性能显著低于其他方法）

Fig. 9 illustrates feature map analysis for instances of large tumors exhibiting internal inhomogeneity. A red dashed box highlights the original input image, while green represents feature maps derived from a 3D CNN module; blue indicates those from a C-LSTM network; purple denotes the output probability map; and the far-right image serves as the manually labeled ground truth.

图形9展示了研究具有内部不均匀性和较大体积特征的大规模肿瘤病例时所进行的特征提取与分析过程。其中红色虚线框标注为输入图像区域，在绿色部分显示的是通过三维CNN模块提取出来的特征图，在蓝色部分则代表C-LSTM网络模块提取出来的特征图。值得注意的是紫色区域展示的是模型输出的概率分布情况，在最右侧图片中则清晰地呈现了人工标注的真实结果分布情况

Fig. 10. 对小肿瘤病例中动脉相位显著增强的特征图进行分析。红虚线框表示输入图像，在3D卷积模块之后提取出绿色特征图，在C-LSTM网络模块之后提取出蓝色特征图，并将紫色输出概率图与右侧的手工标注的标准图像进行对比。

研究图10显示，在动脉期的小型肿瘤案例中具有明显增强特征的区域进行了详细分析。在右侧输入图像区域用虚线框标出；三维 CNN 模块识别出特征区域并将其标记为绿色；C-LSTM 分析结果被标记为蓝色区域；概率分布通过紫色线条表示；而右侧的手工标注区域展示了真实解剖结构。

Table

表

TABLE I network frame work of the proposed model for tumor segmen tation

表 I提出模型的网络框架，用于肿瘤分割

表格II展示了多种模型在分析实验中的数值结果

表 II在消融实验中各种模型的定量结果

TABLE III quantita results of various models in 2.5d cnn scenario

表 III在 2.5D CNN 场景中各种模型的定量结果

TABLE IV performa comparison with external baseline models

表 IV与外部基线模型的性能比较