VQA文献阅读 Learning Conditioned Graph Structures for Interpretable Visual Question Answering

该研究提出了一种基于图卷积网络的可解释视觉问答(VQA)方法，通过该方法通过通过图像节点表示物体 bounding box,节点间连线表物体联系强度,并通过先验知识增强模型可解释性; 实验结果达到1.8%在VQA2数据集上,展示了该模型的有效性与

该贡献得以确切实现

1.提出一个新的、Interpretable、基于图卷积网络的VQA方法
图中的节点表示Image features中的Bounding box ,节点之间的线条表示image中各个节点的联系强度(联系越强，线条越粗)。
线条的学习中，引入了先验知识----问题信息
2.模型的可解释性
通过Image上的bounding box 和 edges 之间的关联，来展示模型的可解释性
3.实验结果
66.18% on VQAv2数据集

3.网络结构

1.We develop a deep neural network that combines spatial, image and textual features in a novel manner in order to answer a question about an image.
2.Our graph learning module then learns an adjacency matrix of the image objects that is conditioned on a given question
3.the spatial graph convolutions - to focus not only on the objects but also on the object relationships that are the
most relevant to the question

4.方法步骤

Step 1.计算模型输入
1.embedding images and questions
Images----(object detecor)---->visual features(bounding box)----->embedding for each bounding box(mean of the corresponding area of the convolutional feature map.)
Questions----(pre-trained word embeddings,e.g GLOVE )------>variable length sequence of embeddings+RNN(GRU) ----Encoded–> single questions embeddings q

Step 2.图网络学习器(基于问题 生成输入图像的图结构表示)
1.图网络学习器概述—学习基于问题的 最相关的邻居节点

该句已符合要求，无需改动

Step 3.空间图卷积(在Step2中得到的questions specific图结构上添加spatial information )
1.coordinate system
前人：for each vertex i, a coordinate system centred at i, with u(i, j) being the coordinates of vertex j in that system
本文：function u(i, j) returns a polar coordinate vector (ρ, θ), describing the relative spatial positions of the centres of the bounding boxes associated with vertices i and j
2.patch operator
前人：

本文：

Step

该步骤涉及损失

5.启示

1.Questions often require answers that cannot be found in the predefifined answers
2.scalar edge weights may not be able to capture the full complexity of the relationships between graph items