Visual Question Answering Using Multimodal Transformers for Post Disaster Assessment

Payyappilly, Leanda J. (2023) Visual Question Answering Using Multimodal Transformers for Post Disaster Assessment. Masters thesis, Indian Institute of Technology Hyderabad.

[img] Text
CE20MTECH12003.pdf - Submitted Version
Restricted to Registered users only until 29 June 2028.

Download (73MB) | Request a copy


Visual Question Answering (VQA) is a challenging multimodal task in the field of artificial intelligence with the goal of answering a question that is related to an image. It is a complex task that requires understanding of both natural language processing (NLP) and computer vision. VQA has its applications in various fields such medicine, video surveillance, for visually impaired personals, etc. The proposed work tries to extend the application of VQA towards post disaster assessment. In the aftermath of natural disasters such as floods, hurricanes, and earthquakes, VQA can play a critical role in aiding disaster relief efforts by fully grasping the situation from post disaster images and offering question- related responses. As of now, research in this field is still progressing and there is a need for developing good quality datasets as well as accurate algorithms need to be created. This work focuses on creating a VQA model using multimodal transformer fusion technique combining the different pretrained text and image transformer models available. The model performance is analyzed using the FloodNet dataset, a High-Resolution Aerial Imagery Dataset for Post Flood Scene Understanding. Also, an attempt is made to train the model on the Ida-BD dataset, a high-resolution satellite imagery for building damage assessment from Hurricane Ida.

[error in script]
IITH Creators:
IITH CreatorsORCiD
Item Type: Thesis (Masters)
Uncontrolled Keywords: VQA, Transformers, Artificial Neural Networks MTD3276
Subjects: Civil Engineering
Civil Engineering > Geoenvironmental Engineering
Divisions: Department of Civil Engineering
Depositing User: Ms Nishitha Prem
Date Deposited: 19 Jul 2023 11:12
Last Modified: 19 Jul 2023 11:12
Publisher URL:
Related URLs:

    Actions (login required)

    View Item View Item
    Statistics for RAIITH ePrint 11524 Statistics for this ePrint Item