Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response (2020-04-14T00:00:00.000000Z)

TL;DR

This paper utilizes convolutional neural networks to define a multimodal deep learning architecture with a modality-agnostic shared representation of social media data to learn a joint representation using state-of-the-art deep learning techniques.

Abstract

Multimedia content in social media platforms provides significant information during disaster events. The types of information shared include reports of injured or deceased people, infrastructure damage, and missing or found people, among others. Although many studies have shown the usefulness of both text and image content for disaster response purposes, the research has been mostly focused on analyzing only the text modality in the past. In this paper, we propose to use both text and image modalities of social media data to learn a joint representation using state-of-the-art deep learning techniques. Specifically, we utilize convolutional neural networks to define a multimodal deep learning architecture with a modality-agnostic shared representation. Extensive experiments on real-world disaster datasets show that the proposed multimodal architecture yields better performance than models trained using a single modality (e.g., either text or image).

Authors

Firoj Alam

5 papers

Ferda Ofli

4 papers

Muhammad Imran

7 papers

TL;DR

Abstract

Authors

References34 items

Multimodal Analysis of Disaster Tweets

Automatic classification of speech overlaps: Feature representation and algorithms

CrisisMMD: Multimodal Twitter Datasets from Natural Disasters

Graph Based Semi-supervised Learning with Convolution Neural Networks to Classify Crisis Related Tweets

Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching

Processing Social Media Images by Combining Human and Machine Computing during Crises

SemEval-2017 Task 4: Sentiment Analysis in Twitter

Damage Assessment from Social Media Imagery Data During Disasters

Image4Act: Online Social Media Image Processing for Disaster Response

Robust Classification of Crisis-Related Data on Social Networks Using Convolutional Neural Networks

Automatic Image Filtering on Social Networks Using Deep Learning and Perceptual Hashing During Crises

Visual Representations of Disaster

How Transferable Are CNN-Based Features for Age and Gender Classification?

Rapid Classification of Crisis-Related Data on Social Networks using Convolutional Neural Networks

Fusing Audio, Textual, and Visual Features for Sentiment Analysis of News Videos

Fusing audio, visual and textual clues for sentiment analysis from multimodal content

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

How transferable are features in deep neural networks?

Going deeper with convolutions

Very Deep Convolutional Networks for Large-Scale Image Recognition

Processing Social Media Messages in Mass Emergency

Fusion of acoustic, linguistic and psycholinguistic features for Speaker Personality Traits recognition

Understanding and classifying image tweets

Efficient Estimation of Word Representations in Vector Space

ADADELTA: An Adaptive Learning Rate Method

ImageNet classification with deep convolutional neural networks

Multimodal Deep Learning

ImageNet: A large-scale hierarchical image database

Combining Pattern Classifiers: Methods and Algorithms

Damage Identification in Social Media Posts using Multimodal Deep Learning

Mining and Classifying Image Posts on Social Media to Analyse Fires

“Together We Stand—Supporting Decision in Crisis Response: Artiﬁcial Intelligence for Digital Response and MicroMappers”

Investigating images as indicators for relevant social media messages in disaster management

Dropout: a simple way to prevent neural networks from overfitting

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names