Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction (2018-09-01T00:00:00.000000Z)

TL;DR

A model that maps raw visual observations to goals using LINGUNET, a language-conditioned image generation network, and then generates the actions required to complete them is designed.

Abstract

We propose to decompose instruction execution to goal prediction and action generation. We design a model that maps raw visual observations to goals using LINGUNET, a language-conditioned image generation network, and then generates the actions required to complete them. Our model is trained from demonstration only without external resources. To evaluate our approach, we introduce two benchmarks for instruction following: LANI, a navigation task; and CHAI, where an agent executes household instructions. Our evaluation demonstrates the advantages of our model decomposition, and illustrates the challenges posed by our new benchmarks.

Authors

Yoav Artzi

13 papers

Dipendra Misra

2 papers

Andrew Bennett

2 papers

TL;DR

Abstract

Authors

References50 items

Scheduled Policy Optimization for Natural Language Communication with Intelligent Agents

Speaker-Follower Models for Vision-and-Language Navigation

Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation

CHALET: Cornell House Agent Learning Environment

IQA: Visual Question Answering in Interactive Environments

Embodied Question Answering

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments

Where is Misty? Interpreting Spatial Descriptors by Modeling Regions in Space

Representation Learning for Grounded Spatial Reasoning

Gated-Attention Architectures for Task-Oriented Language Grounding

Grounded Language Learning in a Simulated 3D World

Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

Modeling Context in Referring Expressions

Instance Normalization: The Missing Ingredient for Fast Stylization

Generation and Comprehension of Unambiguous Object Descriptions

What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment

Grounding English Commands to Reward Functions

Environment-Driven Lexicon Induction for High-Level Instructions

High-Dimensional Continuous Control Using Generalized Advantage Estimation

U-Net: Convolutional Networks for Biomedical Image Segmentation

Adam: A Method for Stochastic Optimization

ReferItGame: Referring to Objects in Photographs of Natural Scenes

Learning Compact Lexicons for CCG Semantic Parsing

Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions

What Are You Talking About? Text-to-Image Coreference

Playing Atari with Deep Reinforcement Learning

Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World

Weakly Supervised Learning of Semantic Parsers for Mapping Instructions to Actions

Unsupervised PCFG Induction for Grounded Language Learning with Highly Ambiguous Supervision

A Joint Model of Language and Perception for Grounded Attribute Learning

Learning to Interpret Natural Language Navigation Instructions from Observations

Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Rectified Linear Units Improve Restricted Boltzmann Machines

Reinforcement learning of motor skills with policy gradients

Walk the Talk: Connecting Language, Knowledge, and Action in Route Instructions

Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping

Intra-Option Learning about Temporally Abstract Actions

Long Short-Term Memory

Expanding the Scope of the ATIS Task: The ATIS-3 Corpus

The Hcrc Map Task Corpus

The ATIS Spoken Language Systems Pilot Corpus

The model uses a convolution neural network for encoding the visual observations, a recurrent neural network with LSTM units to encode the instruction, and a feed-forward network

Chris Apps, Demis Hassabis, and Phil Blunsom

Towards a Dataset for Human Computer Communication via Grounded Language Acquisition

Dropout: a simple way to prevent neural networks from overfitting

Rectifier Nonlinearities Improve Neural Network Acoustic Models

Learning to Parse Natural Language Commands to a Robot Control System

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Gradient-based learning applied to document recognition

Field of Study

Venue Information

Name

Type

URL

Alternate Names