Fast and Continual Knowledge Graph Embedding via Incremental LoRA（三）

阅读量：

Liu et al. present a method for efficient and continuous knowledge graph embedding through incremental LoRA mechanisms. Their work was published as a preprint on arXiv with the identifier arXiv:2407.05705 in the year 2024. The research has been cited by 10 sources.

图1: CKGE 的 IncLoRA 示例图表。随着故事情节的发展，在各个快照中，则展示了具有增量 LoRAs（Low-Rank Approximation）的知识表示引擎（KGE）模型。

3 Methodology

3.1 Preliminary and Problem Statement

Growing Knowledge Graph.

增长的知识图谱（KG）以快照序列的方式表示为G={S₀,S₁,…,Sₙ}。每个快照Sᵢ由一个三元组（Eᵢ,Rᵢ,Tᵢ）构成，在此框架下,Eᵢ代表时间点对应的实体集,Rᵢ是对应的关系集,Tᵢ则包含了所有相关的时间点及其对应的三元组集合。此外,我们定义了各层之间的差异项:将ΔTᵢ=Tᵢ − T_{i−1},ΔEᵢ=Eᵢ − E_{i−1},ΔRᵢ=Rᵢ − R_{i−1}分别作为新生成的时间段内的差异项,这些差异项分别对应着新增的一批三元组、实体更新以及关系集合的变化

Continual Knowledge Graph Embedding.

持续知识图谱嵌入（CKGE）**的目标是将实体与关系嵌入到不断扩展的知识图谱G = {S₀, S₁,…, Sₙ}中。具体而言，在时间点i出现新的三元组∆Tᵢ时，CKGE通过学习新实体∆Eᵢ及其关联关系∆Rᵢ的表示，并更新原有实体Eᵢ₋₁及关系Rᵢ₋₁的表示来适应这一变化。最终系统能够获取到所有相关实体与关系的完整表示信息。

3.2 Framework

FastKGE这一架构通过图2进行展示。总体来说，在知识图谱（KG）各快照不断扩展的过程中，在每一轮新旧知识交替中采用不同层次的增量低秩适配器（LoRAs），以此实现对新增实体与关系的有效学习与存储。

首先，在知识图谱分层阶段，在基于新实体与旧图谱之间的距离以及节点度的基础上进行分类整理；
其次，在IncLoRA学习阶段，在采用具有自适应秩分配的增量LoRAs的方式下完成每层实体及关系的学习任务；

在最后一步骤中，在 link prediction stage, 将所有的 novel LoRA module 聚合成一个 LoRA group, 然后连接所有的 LoRA group 和 initial embedding 来进行推理过程

图2:FastKGE框架概述。LoRA组i表示快照i中所有LoRA的集合。

3.3 Graph Layering

为了实现知识图谱的分层存储，并根据层次给定LoRAs不同的重要性，在知识图谱中新增的知识将按照层级结构进行分类处理。在快照i（i>0）中引入的新三元组∆Ti随后生成新的实体∆Ei与关系∆Ri，并基于旧知识图的距离度量及其中心性评估来确定新增实体的重要性。具体而言，在Si−1的基础上采用广度优先搜索（BFS）方法从Si−1开始逐步扩展快照i中的新增实体序列，并完成排序后的实体序列提取工作

其中，对于ej， ek∈

当j小于或等于k时，在旧图中ej的距离比ek更近。我们将其表示为e在由新的三元组∆Ti组成的新图中的度中心性，并定义为此处的度中心性。

其中fneighbor(e)表示e在∆Ti中的邻居数目。对于与旧图具有相同距离的实体而言，在采用fdc方法进行排序的基础上进一步优化

按重要性排序的实体平均分成N个不同的层

设集合 $E = \{E_1, E_2, \dots, E_N\}$ ，其中 $E_k$ 表示第 $k$ 层实体， $N$ 为超参数。对于关系部分，则采取将所有新增的关系集中放置于一个单独的 $R$ 层中而非分层次存储的方式进行处理。这是因为，在知识图谱的知识动态演化过程中（即随着数据量的增长），实际观察到的是实体数量的增长幅度远高于关系数量的增长幅度，并且在实践中发现总嵌入参数规模与实体数量呈现线性比例关系[Chen等人, 2023]。本文着重研究新实体存储策略及训练方法优化问题，并最终实现了分层次组织的实体与关系结构

和

。

3.4 IncLoRA Learning

Incremental Low-Rank Decomposition

为了提高学习效率并降低训练成本，在本研究中我们提出了一种基于图分层的增量式低秩适配器学习机制IncLoRA。在图分层过程中获得实体与关系的层次结构表示E与R，在此基础上我们实现了对各层次结构中的实体与关系逐层构建增量低秩适配器模型来实现知识的有效存储与更新能力。其中具体而言以其中某一层为例分析其嵌入表示过程：其中某一层k对应的嵌入表示为向量Ek；为了构建该嵌入矩阵Ak∈ℝ^{{n×r}与变换矩阵Bk∈ℝ}{r×d}满足以下约束条件：

其中n表示Ek层中实体的数量,d代表嵌入维度,r则是（Ak,Bk）这对增广矩阵的秩。我们用（Ak,Bk）来表示这一对增量LoRA结构,其目的是为了使低秩学习所涉及的参数数量少于传统训练方法的情况,从而能够获得更快的学习效率,这要求分解过程必须满足特定条件：

采用该方法后，将Ek与Ak和Bk进行整合以形成一个新的增量LORA。最终，在分析∆Ei的所有嵌入时，我们形成了一个LORA组件GEi。

其中concat（·）被用来表示多个矩阵的拼接操作。通过与类似的方法进行分解处理后，我们可以获得一种用于处理新关系类型的学习率调节矩阵群GR i

Adaptive Rank Allocation

为了以保留重要实体的关键信息为主，在不同层次上实施LoRAs优化策略时会采用自适应分级方法

那么

中第k个LoRA的rk为：

最后，确定

中所有适配器的排序。由于R只有一层，因此对于

没有采用自适应的秩分配

IncLoRA Training

由此得到LoRA群

和

在模型训练过程中，并根据需求动态确定每个LoRA（Low-Rank Adaptation）的秩。那么，在知识图谱中，所有的实体Eall和所有的关系Rall都可以通过相应的嵌入形式进行表示。

其中Eorigin和Rorigin分别标识快照0中的实体与关系的原始嵌入向量。最后我们利用新的三元组∆T对GE i与GR i的所有参数进行微调。我们以TransE [Bordes et al. ， 2013]为基础构建知识图谱嵌入（Knowledge Graph Embedding）框架，并采用如下损失函数：

其中 (h ', r, t ') 是 (h, r, t) ∈ ∆Ti 的负三元组，在 TransE 模型中其 scoring function 被定义为 f(h, r, t) = |h + r−t|L1/L2。这里 h ∈ Eall、r ∈ Rall 和 t ∈ Eall 分别代表 h、r 和 t 的嵌入空间中的向量表示，在学习过程中小规模地优化这些嵌入以最小化对应的损失函数。

和

在该模型中，并约束其余所有LoRA组件以及原始嵌入层的参数。最终实现完整的实体与关系表示的具体内容

3.5 Link Predicting

在链接预测阶段中,公式9及10被用来整合所有的LoRA组件。以链路预测为例说明,我们会冻结所有 LoRA 组件及其初始嵌入参数,以便进行后续计算。对于给定的查询（h, r, ?）,通过推导出尾部实体 t 我们能够获得最高的 TransE 评分作为预测结果。值得注意的是,FastKGE 仅涉及在推理阶段之前构建一个完整的嵌入架构,无需额外操作,**因此在整个推理过程中不会产生时间消耗。

References

Ansuini et al., 2019. The paper by Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan explores the essence of data representation dimensions within deep neural networks. It was presented at NeurIPS, in 2019.

[Bordes et al., 2013] Antoine Bordes及其团队于2013年在_NIPS_会议上发表了题为《通过翻译嵌入建模多关系数据》的论文

Antoine Bordes et al., along with their colleagues Jason Weston and Nicolas Usunier, developed a method for open question answering using weakly supervised embedding models. This work was presented in the proceedings of ECML-PKDD in 2014.

Ref. Brown et al., 2020] Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. Language models demonstrate strong capabilities in few-shot learning. Presented at NeurIPS 2020.

A novel approach to entity-agnostic representation learning in knowledge graph embedding with parameter efficiency has been presented by Mingyang Chen et al. in the proceedings of AAAI, 2023. The study focuses on advancing techniques for efficient knowledge graph embedding through innovative entity-agnostic representation learning methods.

[Cui et al., 2023] Yuanning Cui and his team co-authored a paper titled "Lifelong embedding learning and transfer for growing knowledge graphs." Their work was presented at the AAAI conference in 2023.

[Daruna et al. , 2021] Angelina Daruna, Mohan Sridharan, Manju Ramesh, and Svetlana Ivanova. Continuous acquisition of knowledge graph representations. IEEE Robotics and Automation Letters, 6(2):1128–1135, 2021.

The team consisting of Tim Dettmers et al., who presented their work on two-dimensional convolutional knowledge graph embeddings at the AAAI conference in 2018.

[Wenbin et al. , 2014] Wenbin Chen, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. Web-scale probabilistic knowledge fusion through the knowledge vault. In SIGKDD , 2014.

[Hamaguchi et al. , 2017] Takuo Hamaguchi, Hidekazu Oiwa, Masashi Shimbo, and Yuji Matsumoto. 研究超越知识库实体的知识转移作用：采用图神经网络的方法. In IJCAI , 2017.

The work by [Hu et al., 2022] introduces LoRA as a low-rank-based adaptation method for large language models. This contribution was presented in the proceedings of the ICLR conference in 2022.

在2018年发表于

[Kirkpark and others, 2017] James Kirkpark and Razvan Pascanu,... Tiago

Ramalho et al. Circumventing catastrophic loss of learning capacity in deep learning models. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.

该研究团队于[Kou等人（Xiaoyu Kou等）在《Disentangle-based continual graph representation learning》一文中]于2020年发表在EMNLP会议上

[Li et al., 2022] Guozheng Li, Xu Chen, Peng Wang, Jiafeng Xie, and Qiqing Luo. FastRE: Aiming at rapidly extracting relation information through a convolutional encoder-based improved cascade binary tagging framework in IJCAI in the year 2022.

Li et al., 2024] Guozheng Li, Wenjun Ke, Peng Wang, Zijie Xu, Ke Ji, Jiajun Liu, Ziyu Shang, and Qiqing Luo. Advancing instructive in-context learning through the development of tabular prompt engineering techniques for the extraction of relational triple extraction tasks. (arXiv preprint)以及其唯一标识符(arXiv:2402.13741)，发布年份为(Year).

[Liu和她的同事, 2020] 刘 Yuping, Wang Peng, Li Yintai, Shao Yiyan,和Xu Zhongkai. 基于伪残差连接的知识图谱嵌入注意力机制. 在 "COLING" , 2020.

[Liu et al., 2023] Jiajun Liu, Peng Wang, Ziyu Shang, and Chenxiao Wu. Iterde: a cycle-based knowledge distillation framework for knowledge graph embeddings. In AAAI, 2023.

[Liu et al., 2024] Researchers such as Jiajun Liu, Wenjun Ke, Peng Wang, Ziyu Shang, Jinhua Gao, Guozheng Li, Ke Ji and Liu Yanhe aimed to advance incremental knowledge graph embedding through distillation techniques. Their research was shared at the AAAI conference in 2024.

其实在2017年

(Lopez-Paz and Ranzato, 2017) David Lopez-Paz and Marc’Aurelio Ranzato introduced the gradient episodic memory mechanism in their work on continual learning. The paper titled "Gradient episodic memory for continual learning" was published in the proceedings of NIPS, 2017.

出版年份：2021年，在 EMNLP 会议中发表于张哲与王鹏合著的论文《基于双曲几何层次感知的知识图谱嵌入方法用于链接预测》

[Paszke et al., 2019] Adam Paszke and his team presented PyTorch in an imperative style at the NeurIPS conference in 2019. PyTorch is a high-performance deep learning framework developed by Facebook's AI Research lab. The framework provides an imperative programming interface for deep neural networks while maintaining high computational performance.

[Radford et al. , 2019] Alec Radford and his team explored the capabilities of language models as unsupervised multitask learning systems. The paper was published in the OpenAI blog, volume 1, issue 8, page 9 in year 2019.

[Rossi et al. , 2021] Rossi et al., Andrea Rossi, Denilson Barbosa, Donatella Firmani, Antonio Matinata, and Paolo Merialdo. 知识图谱嵌入技术用于链接预测：一种对比分析研究. ACM Transactions on Knowledge Discovery from Data , 15(2):1–49, 2021.

[Rusu et al., 2016] Andrei A Rusu and others conducted extensive research on progressive neural networks. Their work was published as a preprint on arXiv:1606.04671 in the same year.

(Shang et al., 2023) Among Ziyu Shang and his colleagues, the research team has developed AskRL: A framework for aligning spatial knowledge in open-world knowledge graphs through advanced learning techniques. This innovative approach was presented at the proceedings of the ISWC conference in 2023.

[Shang et al., 2024] Ziyu Shang,… Ontofact: Unveiling its impressive fact-skeleton of LLMs through ontology-driven reinforcement learning. In the proceedings of the AAAI Conference in 2024.

[Song and Park, 2018] Song and Park's research on enriching translation-based knowledge graph embeddings through continual learning was featured in the IEEE Access journal in 2018.

（Suchanek et al., 2007）Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum presented "The Yago Core Knowledge Base: Towards a Semantic Web" at the proceedings of the WWW conference held in…

[Sun et al., 2019] Zhiqiang Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang introduced a method for rotating knowledge graphs in complex space to achieve embedding within the proceedings of the ICLR conference held in 2019.

[Toutanova等人, 2015] Kristina toutanova等人的研究表明，在机器翻译任务中使用多模态学习模型能够显著提高模型性能。研究者包括Kristina toutanova、Danqi Chen、Patrick Pantel、Hoifung Poon、Pallavi Choudhury和Michael Gamon等在内的团队成员共同完成了这项研究工作。他们的研究重点在于开发一种高效的方法来将文本表示为结合文本与知识库嵌入的联合嵌入模型，并在第34届EMNLP会议上进行了展示

[Trouillon et al., 2016] The researchers Theo Trouillon, Johannes Welbl, Sébastien Riedel, Eric Gaussier, and Guillaume Bouchard introduced the concept of complex embeddings for simple link prediction. Their work was presented at ICML in 2016.

[Vrandeciˇ c and Kr ´ otzsch, 2014 ¨ ] Denny Vrandeciˇ c and ´ Markus Krotzsch. Wikidata: a free collaborative knowl- ¨ edgebase. Communications of the ACM , 57(10):78–85, 2014.

[Wang et al., 2017] Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. An in-depth exploration of Knowledge Graph Embedding methodologies alongside their practical implementations. IEEE Transactions on Knowledge Engineering, 29(12):2724–2743, 2017.

[Wang et al., 2019] Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and also William Yang Wang presented their work on sentence embedding alignment for the purpose of lifelong relation extraction during the NAACL conference in 2019.

该研究由Wang及其团队在ArXiv上发布，并于(此处可适当扩展)详细探讨了基于正交子空间的学习方法在语言模型持续学习中的应用。该研究由Wang及其团队在ArXiv上发布，并于详细探讨了基于正交子空间的学习方法在语言模型持续学习中的应用。
维基数据项目官方主页邀请您访问维基数据的知识库，并提供丰富的资料以促进知识共享与合作。

[Xu et al. , 2023] Jiaxin Li, Qian Wang, Yuyang He, and Mingyuan Zhang. An advanced two-stage label rectification methodology for enhancing noisy event extraction accuracy. In DASFAA , 2023.

[Zenke et al., 2017] The research team of Friedemann Zenke, along with Ben Poole and Surya Ganguli, explored the concept of continual learning through synaptic intelligence in their paper published at the International Conference on Machine Learning (ICML) in 2017.

[Zhang et al., 2016] Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei-Ying Ma. Collaborative-based knowledge-base embedding for recommendation systems. Presented at the SIGKDD conference in 2016.

Adaptive budget allocation strategy for parameter-efficient fine-tuning. In the proceedings of the ICLR, 2023.