Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question

Published in

Neural Information Processing Systems(2015)

External Links:

Generate Graph

TL;DR

The mQA model, which is able to answer questions about the content of an image, is presented, which contains four components: a Long Short-Term Memory (LSTM), a Convolutional Neural Network (CNN), an LSTM for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer.

Abstract

In this paper, we present the mQA model, which is able to answer questions about the content of an image. The answer can be a sentence, a phrase or a single word. Our model contains four components: a Long Short-Term Memory (LSTM) to extract the question representation, a Convolutional Neural Network (CNN) to extract the visual representation, an LSTM for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. We construct a Freestyle Multilingual Image Question Answering (FM-IQA) dataset to train and evaluate our mQA model. It contains over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations. The quality of the generated answers of our mQA model on this dataset is evaluated by human judges through a Turing Test. Specifically, we mix the answers provided by humans and our model. The human judges need to distinguish our model from the human. They will also provide a score (i.e. 0, 1, 2, the larger the better) indicating the quality of the answer. We propose strategies to monitor the quality of this evaluation process. The experiments show that in 64.7% of cases, the human judges cannot distinguish our model from humans. The average score is 1.454 (1.918 for human). The details of this work, including the FM-IQA dataset, can be found on the project page: this http URL

Authors

Zhiheng Huang

4 papers

W. Xu

7 papers

Jie Zhou

10 papers

References46 items

Image Question Answering: A Visual Semantic Embedding Model and a New Dataset

Exploring Models and Data for Image Question Answering

Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images

VQA: Visual Question Answering

Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images

Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question

Published in

Neural Information Processing Systems(2015)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Zhiheng Huang

4 papers

W. Xu

7 papers

Jie Zhou

10 papers

References46 items

Image Question Answering: A Visual Semantic Embedding Model and a New Dataset

Exploring Models and Data for Image Question Answering

Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images

VQA: Visual Question Answering

Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images

Haoyuan Gao

1 papers

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Learning Longer Memory in Recurrent Neural Networks

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Learning From Weakly Supervised Data by The Expectation Loss SVM (e-SVM) algorithm

Deep visual-semantic alignments for generating image descriptions

Simple Image Description Generator via a Linear Phrase-Based Approach

Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation

CIDEr: Consensus-based image description evaluation

Learning a Recurrent Visual Representation for Image Caption Generation

From captions to visual concepts and back

Show and tell: A neural image caption generator

Long-term recurrent convolutional networks for visual recognition and description

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Explain Images with Multimodal Recurrent Neural Networks

A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input

Going deeper with convolutions

Sequence to Sequence Learning with Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Microsoft COCO: Common Objects in Context

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Distributed Representations of Words and Phrases and their Compositionality

Recurrent Continuous Translation Models

Joint Video and Text Parsing for Understanding Events and Answering Queries

ImageNet classification with deep convolutional neural networks

VizWiz: nearly real-time answers to visual questions

Rectified Linear Units Improve Restricted Boltzmann Machines

METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

Bleu: a Method for Automatic Evaluation of Machine Translation

Long Short-Term Memory

Verb Semantics and Lexical Selection

Finding Structure in Time

Computing Machinery and Intelligence

Long-Term Recurrent Convolutional Networks for Visual Recognition and Description

A visual Turing test for computer vision systems

Efficient BackProp

Recurrent neural network based language model

The IAPR TC-12 Benchmark: A New Evaluation Resource for Visual Information Systems

Field of Study

Computer Science

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS