Grounded Language Learning in a Simulated 3D World (2017-06-20T00:00:00.000000Z)

TL;DR

An agent is presented that learns to interpret language in a simulated 3D environment where it is rewarded for the successful execution of written instructions and its comprehension of language extends beyond its prior experience, enabling it to apply familiar language to unfamiliar situations and to interpret entirely novel instructions.

Abstract

We are increasingly surrounded by artificially intelligent technology that takes decisions and executes actions on our behalf. This creates a pressing need for general means to communicate with, instruct and guide artificial agents, with human language the most compelling means for such communication. To achieve this in a scalable fashion, agents must be able to relate language to the world and to actions; that is, their understanding of language must be grounded and embodied. However, learning grounded language is a notoriously challenging problem in artificial intelligence research. Here we present an agent that learns to interpret language in a simulated 3D environment where it is rewarded for the successful execution of written instructions. Trained via a combination of reinforcement and unsupervised learning, and beginning with minimal prior knowledge, the agent learns to relate linguistic symbols to emergent perceptual representations of its physical surroundings and to pertinent sequences of actions. The agent's comprehension of language extends beyond its prior experience, enabling it to apply familiar language to unfamiliar situations and to interpret entirely novel instructions. Moreover, the speed with which this agent learns new words increases as its semantic knowledge grows. This facility for generalising and bootstrapping semantic knowledge indicates the potential of the present approach for reconciling ambiguous natural language with the complexity of the physical world.

Authors

References41 items

Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularities

A Deep Compositional Framework for Human-like Language Acquisition in Virtual Environment

Learning From Explanations Using Sentiment and Advice in RL

Reinforcement Learning with Unsupervised Auxiliary Tasks

Learning Language Games through Interaction

Asynchronous Methods for Deep Reinforcement Learning

A Roadmap Towards Machine Intelligence

Order-Embeddings of Images and Language

Action-Conditional Video Prediction using Deep Networks in Atari Games

Learning to Interpret Natural Language Commands through Human-Robot Dialog

Language Understanding for Text-based Games using Deep Reinforcement Learning

Human-level control through deep reinforcement learning

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Learning spatial-semantic representations from natural language descriptions and scene classifications

A framework for learning semantic maps from grounded natural language descriptions

Semantic Parsing on Freebase from Question-Answer Pairs

Grounded Language Learning from Video Described with Sentences

ImageNet classification with deep convolutional neural networks

Grounded Models of Semantic Representation

Learning to sportscast: a test of grounded language acquisition

Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars

Semantic Cognition: A Parallel Distributed Processing Approach

Grounding the Lexical Semantics of Verbs in Visual Perception using Force Dynamics and Event Logic

Long Short-Term Memory

Naming in young children: a dumb attentional mechanism?

Grounding language in perception

Evidence for Representations of Perceptually Similar Natural Categories by 3-Month-Old and 4-Month-Old Infants

Mental models of the earth: A study of conceptual change in childhood

Backpropagation Applied to Handwritten Zip Code Recognition

Language learnability and language development

A Review of B. F. Skinner's Verbal Behavior

Colours

Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude