TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018-11-29T00:00:00.000000Z)

TL;DR

This work introduces the Touchdown task and dataset, where an agent must first follow navigation instructions in a Street View environment to a goal position, and then guess a location in its observed environment described in natural language to find a hidden object.

Abstract

We study the problem of jointly reasoning about language and vision through a navigation and spatial reasoning task. We introduce the Touchdown task and dataset, where an agent must first follow navigation instructions in a Street View environment to a goal position, and then guess a location in its observed environment described in natural language to find a hidden object. The data contains 9326 examples of English instructions and spatial descriptions paired with demonstrations. We perform qualitative linguistic analysis, and show that the data displays a rich use of spatial reasoning. Empirical analysis shows the data presents an open challenge to existing methods.

Authors

Noah Snavely

16 papers

Alane Suhr

3 papers

Yoav Artzi

13 papers

TL;DR

Abstract

Authors

References41 items

A Corpus for Reasoning about Natural Language Grounded in Photographs

Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction

Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

On Evaluation of Embodied Navigation Agents

Talk the Walk: Navigating New York City through Grounded Dialogue

Learning to Navigate in Cities Without a Map

CHALET: Cornell House Agent Learning Environment

AI2-THOR: An Interactive 3D Environment for Visual AI

IQA: Visual Question Answering in Interactive Environments

Embodied Question Answering

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments

Weakly Supervised Semantic Parsing with Abstract Examples

Where is Misty? Interpreting Spatial Descriptors by Modeling Regions in Space

A Corpus of Natural Language for Visual Reasoning

Gated-Attention Architectures for Task-Oriented Language Grounding

Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

Learning Deep Representations of Fine-Grained Visual Descriptions

Asynchronous Methods for Deep Reinforcement Learning

Deep Residual Learning for Image Recognition

Generation and Comprehension of Unambiguous Object Descriptions

U-Net: Convolutional Networks for Biomedical Image Segmentation

VQA: Visual Question Answering

Microsoft COCO Captions: Data Collection and Evaluation Server

Adam: A Method for Stochastic Optimization

ReferItGame: Referring to Objects in Photographs of Natural Scenes

ImageNet Large Scale Visual Recognition Challenge

Microsoft COCO: Common Objects in Context

Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits

Bringing Semantics into Focus Using Visual Abstraction

A Joint Model of Language and Perception for Grounded Attribute Learning

Learning to Interpret Natural Language Navigation Instructions from Observations

Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Reinforcement Learning for Mapping Instructions to Actions

Walk the Talk: Connecting Language, Knowledge, and Action in Route Instructions

Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping

Long Short-Term Memory

ABCCNN: An Attention Based Convolutional Neural Network for visual question answering

Towards a Dataset for Human Computer Communication via Grounded Language Acquisition

Function Optimization using Connectionist Reinforcement Learning Algorithms

and as an in

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names