Dual Attention Networks for Multimodal Reasoning and Matching (2016-11-02T00:00:00.000000Z)

TL;DR

This work proposes Dual Attention Networks which jointly leverage visual and textual attention mechanisms to capture fine-grained interplay between vision and language and introduces two types of DANs for multimodal reasoning and matching, respectively.

Abstract

We propose Dual Attention Networks (DANs) which jointly leverage visual and textual attention mechanisms to capture fine-grained interplay between vision and language. DANs attend to specific regions in images and words in text through multiple steps and gather essential information from both modalities. Based on this framework, we introduce two types of DANs for multimodal reasoning and matching, respectively. The reasoning model allows visual and textual attentions to steer each other during collaborative inference, which is useful for tasks such as Visual Question Answering (VQA). In addition, the matching model exploits the two attention mechanisms to estimate the similarity between images and sentences by focusing on their shared semantics. Our extensive experiments validate the effectiveness of DANs in combining vision and language, achieving the state-of-the-art performance on public benchmarks for VQA and image-text matching.

Authors

Hyeonseob Nam

2 papers

Jung-Woo Ha

11 papers

Jeonghee Kim

1 papers

TL;DR

Abstract

Authors

References37 items

Training Recurrent Answering Units with Joint Loss Minimization for VQA

Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Multimodal Residual Learning for Visual QA

Hierarchical Question-Image Co-Attention for Visual Question Answering

Dynamic Memory Networks for Visual and Textual Question Answering

Deep Residual Learning for Image Recognition

Simple Baseline for Visual Question Answering

Where to Look: Focus Regions for Visual Question Answering

Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources

Learning Deep Structure-Preserving Image-Text Embeddings

Image Question Answering Using Convolutional Neural Network with Dynamic Parameter Prediction

Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data

Natural Language Object Retrieval

Neural Module Networks

Stacked Attention Networks for Image Question Answering

A Neural Attention Model for Abstractive Sentence Summarization

Ask Me Anything: Dynamic Memory Networks for Natural Language Processing

Deep correlation for matching images and text

Associating neural word embeddings with deep image representations using Fisher Vectors

A Hierarchical Neural Autoencoder for Paragraphs and Documents

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

VQA: Visual Question Answering

Multimodal Convolutional Neural Networks for Matching Image and Sentence

DRAW: A Recurrent Neural Network For Image Generation

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Deep visual-semantic alignments for generating image descriptions

Show and tell: A neural image caption generator

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Very Deep Convolutional Networks for Large-Scale Image Recognition

Neural Machine Translation by Jointly Learning to Align and Translate

Deep Networks with Internal Selective Attention through Feedback Connections

Recurrent Models of Visual Attention

Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

Microsoft COCO: Common Objects in Context

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names