GuessWhat?! Visual Object Discovery through Multi-modal Dialogue (2016-11-23T00:00:00.000000Z)

TL;DR

This work introduces GuessWhat?

Abstract

We introduce GuessWhat?!, a two-player guessing game as a testbed for research on the interplay of computer vision and dialogue systems. The goal of the game is to locate an unknown object in a rich image scene by asking a sequence of questions. Higher-level image understanding, like spatial reasoning and language grounding, is required to solve the proposed task. Our key contribution is the collection of a large-scale dataset consisting of 150K human-played games with a total of 800K visual question-answer pairs on 66K images. We explain our design decisions in collecting the dataset and introduce the oracle and questioner tasks that are associated with the two players of the game. We prototyped deep learning models to establish initial baselines of the introduced tasks.

Authors

H. Larochelle

18 papers

Aaron C. Courville

35 papers

O. Pietquin

6 papers

TL;DR

Abstract

Authors

References51 items

Modeling Context in Referring Expressions

Learning Aligned Cross-Modal Representations from Weakly Aligned Data

Analyzing the Behavior of Visual Question Answering Models

Hierarchical Question-Image Co-Attention for Visual Question Answering

A Network-based End-to-End Trainable Task-oriented Dialogue System

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

A Survey of Available Corpora for Building Data-Driven Dialogue Systems

Where to Look: Focus Regions for Visual Question Answering

Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

Natural Language Object Retrieval

Generation and Comprehension of Unambiguous Object Descriptions

Stacked Attention Networks for Image Question Answering

Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models

Hierarchical Neural Network Generative Models for Movie Dialogues

VQA: Visual Question Answering

Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Adam: A Method for Stochastic Optimization

Learning Deep Features for Scene Recognition using Places Database

Deep visual-semantic alignments for generating image descriptions

Show and tell: A neural image caption generator

ReferItGame: Referring to Objects in Photographs of Natural Scenes

A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input

Sequence to Sequence Learning with Neural Networks

ImageNet Large Scale Visual Recognition Challenge

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Microsoft COCO: Common Objects in Context

POMDP-Based Statistical Spoken Dialog Systems: A Review

A survey on metrics for the evaluation of user simulations

Data-Driven Methods for Adaptive Spoken Dialogue Systems

Computational Generation of Referring Expressions: A Survey

Software Framework for Topic Modelling with Large Corpora

The segmented and annotated IAPR TC-12 benchmark

Et al

A probabilistic framework for dialog simulation and optimal strategy learning

Dynamic topic models

Peekaboom: a game for locating objects in images

One-shot learning of object categories

Labeling images with a computer game

Reinforcement Learning for Spoken Dialogue Systems

Long Short-Term Memory

A stochastic model of computer-human interaction for learning dialogue strategies

and L

Deep Learning

A visual Turing test for computer vision systems

editor

Amazon’s Mechanical Turk a new source of inexpensive, yet high-quality, data?

20 questions.

DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies

Visual properties We group objects by their size – Is it big? , shape – Is it square? – or color – Is it blue?

Object taxonomy We can use the hierarchical structure of object categories, i.e. taxonomy, to group objects e.g. Is it a vehicle? to refer to both cars and trucks. to locate different in of the

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names