MIST：用于组织病理学亚型预测的多实例选择性Transformer|文献速递--基于深度学习的医学影像病灶分割

Title

题目

MIST: A multi-instance attention mechanism designed to classify histopathological subtypes.

MIST：用于组织病理学亚型预测的多实例选择性Transformer

01

文献速递介绍

在癌症疾病的诊断和治疗中具有关键的临床意义的组织病理学亚型预测，在医学领域发挥着不可替代的作用。这种预测技术的目标是通过分析全视野图像（WSI）中的病变区域来鉴别不同的病理类型（图1(1)），例如包括正常黏膜、碎屑细胞、病理性良性组织、淋巴细胞以及侵袭性癌等情况（Han等, 2022）。该技术通常建立在临床医生的专业判断基础之上，并以此为基础制定最优治疗方案（Han等, 2017）。深入了解组织病理学亚型的相关知识有助于医学生物学家早期识别肿瘤转移的机会并据此制定个性化的治疗计划（Han等, 2017），这在乳腺癌、结直肠癌等多种癌症类型中均得到了应用。此外，在肿瘤微环境研究方面取得的新突破也为临床效果评估提供了有力支持（Gurcan等, 2009；Kather等, 2019, 2018, 2017）。

Abatract

摘要

Accurate histopathological subtype prediction is of great clinical significance for cancer diagnosis and tumor microenvironment analysis, yet it presents a formidable challenge due to (1) the instance-level discrimination of histopathological images, (2) the significant variations in shape and chromatin texture among images from different classes and within the same class, and (3) the heterogeneity in feature distribution across images. In this study, we tackle subtype prediction by framing it as a fine-grained representation learning problem and introduce a novel multi-instance selective transformer (MIST) framework. This approach effectively achieves accurate histopathological subtype prediction by incorporating a refined mechanism for instance identification. The MIST framework integrates an innovative selective self-attention mechanism that combines multi-instance learning (MIL) with vision transformers (ViT), enabling the adaptive identification of representative instances for detailed representation. Each instance is granted distinct contributions to the bag-level representation through its interactions with other instances and bags. Specifically, the SiT module employs selective multi-head self-attention (S-MSA) to identify key instances based on pairwise interactions between them. Complementarily, the MIFD module employs an information bottleneck strategy to learn discriminative fine-grained representations for histopathological images by modeling interactions between selected instances and bags. Extensive experiments conducted on five clinical benchmarks demonstrate that our MIST framework achieves accurate histopathological subtype prediction and attains state-of-the-art performance with an accuracy of 0.936. Furthermore, our method shows promising potential for advancing fine-grained medical image analysis, particularly in applications such as histopathological subtype prediction in clinical settings.

在临床实践中，精确的组织病理学亚型划分对于癌症诊断及肿瘤微环境研究具有重要意义。鉴于以下三个主要挑战，在实现这一目标方面仍面临诸多困难：(1)图像实例间的清晰区分问题；(2)图像形状与染色质纹理特征间表现出的小类内差异与大类间差异；以及(3)不同样本间的异质性特征分布不均匀性问题。本研究将针对这一领域提出一种创新性的解决方案：我们将亚型预测问题建模为细粒度表征学习任务，并提出了一种基于多实例选择性Transformer（MIST）的新架构框架。该框架通过设计一种结合多实例学习（MIL）与视觉Transformer（ViT）的选择性自注意机制，在自动识别有用图像实例方面展现出显著优势特性。值得注意的是，在MIST模型中引入了一种新的模块化设计思路：通过SiT模块实现了选择性多头自注意力机制（S-MSA），该机制能够有效捕捉并建模不同图像实例之间的相互作用关系；同时又通过信息瓶颈策略设计了MIFD模块，在此过程中实现了所选实例对 bag 表示空间贡献度的差异化表达。具体而言，在所开发的 MIST 模型中：首先通过 SiT 模块实现了选择性自注意力机制；其次又结合了 MIFD 模块，在此过程中实现了所选实例对 bag 表示空间贡献度的差异化表达。通过五个临床基准数据集上的大量实验测试表明：该方法不仅能够实现精确的组织病理学亚型预测目标，并且在性能指标上达到了 0.936 的高准确率水平；同时该方法还展现了显著的优势特性——能够在细粒度医学图像分析领域取得突破性进展

Method

方法

The novelly developed MIST (as shown in Fig. 3) represents histopathological subtype prediction as a form of fine-grained representation learning and realizes this through the construction of a multiple-stage vision transformer enhanced by multi-instance learning. The novelly developed MIST enables the extraction of instance-level fine-grained features by adaptively allocating varying levels of contribution from each individual instance to the histopathological image representation. Consequently, this advanced framework comprises three closely integrated components:

(1) The discriminant instance-transformer (DIT), augmented with a discriminant self-attention module (DSAM), is designed to selectively extract characteristic instances from a given bag, which are then used to create a discriminative representation of histopathological images. The DIT selects instances by modeling their interactions through a selective self-attention mechanism; (2) The hierarchical multiple-instance feature decoding (HMIFD) method is developed to systematically acquire high-resolution instance-level features for histopathological subtype classification through structured modeling of interactions between individual instances and their parent bags.

(3) A loss function incorporating information bottlenecks is employed to classify histopathological subtypes using a bag representation learned through MIST, which integrates both instance-level and bag-level interactions between instances.

其将组织病理学亚型预测表示为细粒度表征学习，并通过构建多阶段视觉Transformer结合多实例学习实现了该预测任务。
其新设计的MIST通过自适应地赋予每个实例不同的贡献完成了对实例级别细粒度特征的学习。
因此所述MIST模型包含三个紧密相关的组成部分：

(1) 选择性实例Transformer（SiT）：基于选择性自注意机制（S-MSA），该方法根据需求从样本集合中自动筛选出最具代表性的实例集合，并旨在实现组织病理学图像的独特表征特征提取过程。该模型通过构建基于选择性自注意机制的交互式架构来优化实例间的相互作用关系；

多实例特征解耦（MIFD）：通过构建实例与其 Bag 之间的相互作用模型,逐步推导出用于组织病理学亚分类预测的高分辨率级联特征表示;

基于信息瓶颈理论构建损失函数：通过融合实例间的相互作用以及考虑实例与其所属类别间的关联，在MIST算法下提取分类所需的类别特征向量以识别组织病理学亚型分布。

Conclusion

结论

In this study, a novel multi-instance selection transformer framework named MIST is developed to realize histopathological subtype prediction for cancer prognosis. The MIST architecture incorporates both instance-to-bag and instance-to-instance interaction modeling, which is achieved by designing a selective instance transformer (SiT) equipped with selective multi-head self-attention (S-MSA). This innovative approach enables the progressive extraction of critical instance-level features within the context of bag-level representations, thereby facilitating fine-grained image analysis and achieving robust predictive performance. By leveraging bag-level prior knowledge during multiple instance feature decoupling (MIFD), redundancy is minimized, preventing potential performance degradation associated with modeling inter-instance relationships. To address these challenges, an information bottleneck loss function is introduced, guiding the network to learn a compact yet discriminative representation that effectively captures essential features for histopathological image analysis. Experimental results demonstrate that MIST significantly outperforms existing methods in terms of accuracy and robustness for fine-grained histopathological image prediction, offering great potential for clinical cancer diagnosis and prognosis through its advanced computational framework.

在本文中，作者提出了多实例选择性Transformer（MIST）框架，并将其应用于癌症预后中的组织病理学亚型预测任务。该框架设计了一种创新结构，在同时建模实例间及其与其他集合体之间的相互作用方面具有显著优势。具体而言，在提出一种新颖的选择性实例Transformer（SiT）的基础上，并结合了基于选择性的多头自注意力机制（S-MSA），该方法能够逐步提取细粒度的实例级特征以实现高分辨率表征。此外，在模型架构中引入了多实例特征解耦（MIFD）模块，在利用集合体级别的先验知识来缓解冗余问题的同时逐步学习任务相关的表征，并构建了一个信息瓶颈损失函数来引导网络学习最小但充分的判别特征表示。实验结果表明，在组织病理学图像的细粒度预测任务中该方法表现优异，并展现了在临床癌症诊断领域巨大的应用潜力

Figure

图

_Fig. 1. 三个主要挑战阻碍了对准确的 histopathological 子类型预测。（1）对于癌亚型或病理组织来说对它们精细描述的特征表示问题。（2）在形态和染色质纹理方面类别间的差异较小而类内的差异较大这使得识别变得异常困难。（3）在不同图像中 histopathological 特征分布不均的情况下进行统一特征提取可能导致预测性能下降因此需要根据不同的 histopathology 图像自适应地选择高度显著的病变区域以提高预测效果。

图1. 三个障碍影响了准确的组织病理学亚型预测：(1) 对癌细胞或病理组织样本进行细致粒度特征描述存在挑战。(2) 形状与染色质纹理之间两类间的差异较小而同一类内部的差异较大,这些特征非常难以准确识别。(3) 各个图像中存在异质性分布的不同组织病理学特征,若采用统一的方法提取这些特征可能导致预测性能下降,因此需要根据具体情况自适应地选择各个图像中高度重要的病理区域。

Fig. 2. Distinct from existing multi-instance learning (MIL), this new MISTin method innovatively formalizes subtype prediction as a fine-grained representation learning task that integrates MIL and ViT, subsequently developing a novel selective self-attention mechanism to facilitate histopathological subtype prediction. This approach ensures that each instance contributes unequally to the histopathological image's representation through optimized instance selection and feature decoupling.

相较于现有采用多实例学习（MIL）的方法而言，本文提出了一种创新性的新方法，将亚型预测表达为一种结合多实例学习（MIL）与视觉Transformer（ViT）的细粒度表征学习方式，并构建了基于选择性自注意机制的选择性自注意模型来进行组织病理学亚型预测。该方法通过引入实例选择机制及特征解耦技术，在保证模型性能的同时显著提升了计算效率，并实现了各实例对组织病理学图像表征的作用程度存在差异性的动态刻画。

The MIST method formulates histopathological subtype prediction as a fine-grained representation learning problem, introducing an innovative selective self-attention mechanism through the advancement of vision transformer architecture and multi-instance learning (MIL). It progressively learns the fine-grained histopathological representation to achieve histopathological subtype prediction. The MIST system is composed of three core components: (1) a selective instance transformer (SiT) with S-MSA, which dynamically identifies representative instances for histopathological image representation by modeling instance-to-instance interactions via selective self-attention mechanisms; (2) multiple instance feature decoupling (MIFD), a process that gradually extracts instance-level fine-grained features for histopathological subtype prediction by modeling instance-to-bag interactions; (3) an information bottleneck loss function, which trains a multi-stage transformer network by integrating both instance-to-instance and instance-to-bag interaction principles inspired by multi-instance learning.

图3. MIST将组织病理学亚型预测表征化为一种精细粒度的学习问题，并通过优化视觉Transformer架构以及多示例学习（MIL）构建了一种创新的选择性自注意力机制框架。该系统通过系统地构建细粒度组织病理学表征来实现亚类预测目标。其核心模块由三个关键组件构成：首先设计了选择性示例Transformer（SiT），该组件基于选择性自注意力机制模拟示例间的相互作用关系，并能动态筛选出最具代表性的组织病理学图像特征示例；其次开发了多示例特征解耦（MIFD）模块，在分析样本间关联的基础上实现对单个示例级别的精细粒度特征提取；最后引入信息瓶颈损失函数框架，在综合考虑单个样本及其所属类别袋之间相互关系的基础上设计了高效的多示例学习训练策略。

_Fig. 4. (a) 如图4(a)所示，SiT能够有效提取组织病理图像的细粒度特征。(b) S-MSA具有选择性自注意力机制，并由instance scoring和WAIS组成。该模型通过建模实例间的相互作用逐步选择具有代表性的实例来构建细粒度特征。其中(instance scoring)(给每个实例赋予重要性评分)(WAIS)(根据重要性评分自适应地选择具有代表性的实例作为细粒度特征。)**

图4. (a) SiT学习用于组织病理学图像的细粒度表征。(b) S-MSA结合了具有选择性的自注意力机制，并采用基于实例的评分方法结合自适应索引系统WAIS（简称为WAIS）。该系统通过逐步建模各实例间的相互作用关系来筛选出具有代表性的样本作为细粒度特征。其中，在每个样本中引入了基于评分机制的重要性赋值过程以指导特征提取过程。

Fig. 5. The systematic framework MIFD systematically acquires fine-grained histopathological image representations for subtype prediction through integrating discriminative instance-level features into bag representation via an optimized instance-to-bag interaction mechanism. By minimizing the association between these representations and reducing potential negative impacts on task-specific feature extraction, it effectively mitigates performance degradation in histopathological subtype prediction.

图5. MIFD通过将具有判别的实例级特征融合到袋表示中，并结合实例与袋之间的相互作用机制, 学习组织病理学图像中的细致粒度特征, 以实现亚型预测. 该方法降低了袋级表示与输入实例特征中无关联信息的相关性, 从而防止组织病理学亚型预测性能因无关联信息而导致性能下降.

Fig. 6. The curve of training loss indicates substantial enhancements in the training process’s convergence across multiple modules and diverse datasets.

图6展示了训练损失曲线，在多个数据集上表明MIST结合不同模块在训练收敛性方面带来了显著改进。

The visualization results show that MIST combined with WAIS in S-MSA tends to emphasize increasingly significant instances. The dashed box denotes the critical areas in histopathology, whereas masked regions signify those deemed unnecessary and discarded after 12 stages. This phenomenon highlights the sufficient interpretability of S-MSA.

可视化结果显示，在采用S-MSA方法处理WAIS-MIST的过程中，默认逐步聚焦到最具区分度的例子中。具体而言，在图7中可以看到虚线框标注组织学上关键的部位（即具有最高判别价值的例子），而未被关注的部分则是在12个阶段后被认为不具鉴别力并予以舍弃。这种现象进一步揭示了S-MSA在可解释性方面的显著优势。

Among the tested approaches, the S-MSA of MIST demonstrates the highest accuracy in histopathological subtype prediction, which captures discriminative and subtle features. When compared to various instance selection strategies, including fixed and average methods, applied to the BreaKHis and BRACS datasets, S-MSA achieves superior performance in this domain.

图8. 在BreaKHis和BRACS数据集中，相较于多种实例选择方案（如固定及平均实例选择方法），MIST的S-MSA通过捕获关键特征细节，在组织病理学分类问题中实现了最高准确率。

_Fig. 9. Visualization results demonstrate that attributed to the proposed selective self-attention mechanism, the MIST can effectively identify the most informative regions in images, which are characterized as either simple informative regions (located in the first and third rows) or complex informative regions (found in the second and last rows), compared to different instance selection methods. Specifically, the second-column annotated images with dashed polygons correspond to significant histopathological regions.

在图9中展示可视化结果显示结果，并基于所提出的具有选择性自注意力机制的方法进行分析后发现,MIST算法能够有效地识别图像中最具信息量的关键区域,无论这些区域呈现简单特征（如第一与第三行）还是复杂特征（如第二与最后一行）。特别值得注意的是,在第二列使用虚线多边形标注的组织病理学区域,其表现尤为突出

图10展示了具有特殊 histopathological subtype 的病例可视化情况。（a）来自 BRACS 数据集的 7 种不同的 histopathological 子类型图片中，FEA 子类型图片通常在其中心位置具有较大的背景区域。（b）针对 BRACS 测试集中的 7 种不同子类型预测结果进行了展示，并使用柱状图呈现方式。我们的 MIST 方法在处理具有中心背景的特殊子类型病例时实现了第二名的优异结果。

图10展示了特殊组织病理学亚型案例的表现形式。（a）从BRACS数据集中获取了七种不同的组织病理学亚类型的代表性切片图像；FEA类型的切片通常呈现出中心区域具有显著大背景面积的现象。（b）通过柱状图展示了BRACS测试集中七种不同亚型样本预测效果的具体分布情况；特别地，在这些具有中心区域显著大背景面积的关键样本中，“MIST”模型表现出了第二佳的效果。

Table

表

The experiment results in Table 1 demonstrate that MIST obtains outstanding performance in histopathological subtype prediction across five challenging datasets.

如图1所示，在经过一系列具有挑战性的实验测试后，MIST方法在多个领域中表现出了卓越的组织病理学亚类预测性能水平。

Table 2 Performance Analysis of the MIST under various configuration settings for histopathological subtype prediction employing five evaluation criteria across three datasets.

如表所示，在本研究中基于三个数据集采用五个评估指标对MIST系统针对不同配置方案下的组织病理学亚型预测性能水平进行了全面评估。

Table 3 demonstrates that MIST achieves competitive performance when compared to SOTA methods on dataset NCT-CRC-HE. Notes: bold represents SOTA results, whereas underlined denotes second-best outcomes.

表3 显示实验结果, 该方法在NCT-CRC-HE数据集上展现出相当竞争力的表现, 达到了与现有最佳方案相媲美的效果.注: 标记表示最佳结果, 表示第二好结果

Table 4 lists the experimental results, demonstrating that MIST achieves competitive performance when compared to SOTA methods on the BreaKHis dataset. Notes: The bold indicates SOTA results, while underlining highlights the second-best outcomes.

表4 实验结果表明,MIST在BreaKHis数据集中展现了与当前最先进的方法相当有竞争力的表现.注: **表示当前方法表现最佳的结果,_表示次优的结果.

Empirical findings in Table 5 demonstrate that MIST achieves notable performance in comparison with SOTA methods on BRACS dataset. The notes state that this is optimal, whereas a close alternative is also recognized.

从实验结果来看，在BRACS数据集上MIST的表现优于当前最先进方法，并展现出具有竞争力的优势。注释部分显示：**表示当前最优结果；_ _表示次优结果。

Table 6 Experimental results indicate that MIST achieves a superior performance when contrasted against SOTA methods in histopathological subtype analysis derived from private datasets at Fujian Medical University Union Hospital. Notes: The SOTA result is highlighted by bold text, whereas the second-best outcome is emphasized using an underscore.

实验结果表明表6显示，在福建医科大学附属协和医院的私有数据组织病理学亚型数据集中，相较于当前最先进方法展示了具有竞争力的表现。注释显示最突出的结果，并用下划线标注次优的结果

Table 7 demonstrates that MIST achieves a competitive performance against SOTA methods on Camelyon16. Notes: Bold highlights SOTA results, underlining those of second-best.

表7 实验结果表明，在Camelyon16数据集上基于现有方法的研究组的最佳成果（以粗体显示）实现了比现有方法更高的性能（下划线表示次优成果）。

Table 8 Research experiments on S-MSA Sensitivity in various stages within the histopathological classification dataset of private data sourced from Fujian Medical University Union Hospital.

表格8展示了S-MSA在各个时间段内针对福建医科大学附属协和医院私有组织病理学亚型数据集进行的敏感性测试结果

第九表展示了在不同头的部分进行的SMSASensitivity实验结果

表格9中的S-MSA方法在针对不同数量的组织病理学亚型数据集时展示了对福建医科大学附属协和医院私有数据的高度敏感度研究。

Table 10 experiments examining the scalability of various bag sizes and examples on the Camelyon16 dataset.

表10 在Camelyon16数据集上不同袋大小和实例数量的可扩展性实验。

Table eleven Experimental results based on various instance selection approaches in the BreaKHis and BRACS datasets.

表11 不同实例选择策略在BreaKHis和BRACS数据集上的实验结果。

Table 12: Conducted experiments of Lagrange multipliers β in various configurations on the BreaKHis dataset.

表12 在BreaKHis数据集上对不同Lagrange乘数𝛽赋值的实验结果。

Table 13 presents the statistical analysis comparing MIST to the baseline model using a t-test at a significance level of 0.1. If the p-value is lower than 0.001, it suggests that MIST achieves significantly different performance compared to the baseline model.

表13展示了采用t检验方法在α=0.1的显著性水平上比较MIST与基线模型之间的统计差异。研究结果显示p值小于等于0.001时表明MIST与基线模型之间呈现出高度显著差异。