Semantic relation graph reasoning network for visual question answering

Hong Lan; Pufen Zhang

doi:10.1117/12.2588837

20 January 2021 Semantic relation graph reasoning network for visual question answering

Hong Lan, Pufen Zhang

Proceedings Volume 11719, Twelfth International Conference on Signal Processing Systems; 117190J (2021) https://doi.org/10.1117/12.2588837
Event: Twelfth International Conference on Signal Processing Systems, 2020, Shanghai, China

Abstract

In order to answer semantically-complicated questions about an image, a Visual Question Answering (VQA) model needs to fully understand the visual scene in the image, especially the dynamic interaction between different objects. This task inherently requires reasoning the visual relationships among the objects of image. Meanwhile, the visual reasoning process should be guided by the information of the question. In this paper, we proposed a semantic relation graph reasoning network, the process of semantic relation reasoning is guided by the cross-modal attention mechanism. In addition, a Gated Graph Convolutional Network (GGCN) constructed based on cross-modal attention weights that novelly injects the semantic interaction information between objects into their visual features, and the features with relational awareness are produced. In particular, we trained a semantic relationship detector to extract the semantic relationship between objects for constructing the semantic relation graph. Experiments demonstrate that proposed model outperforms most state-of-the-art methods on the VQA v2.0 benchmark datasets.

Citation Download Citation

Hong Lan and Pufen Zhang "Semantic relation graph reasoning network for visual question answering", Proc. SPIE 11719, Twelfth International Conference on Signal Processing Systems, 117190J (20 January 2021); https://doi.org/10.1117/12.2588837

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available