Dream to Control: Learning Behaviors by Latent Imagination

Published in

International Conference on Learning Representa...(2019)

External Links:

Generate Graph

TL;DR

Dreamer is presented, a reinforcement learning agent that solves long-horizon tasks purely by latent imagination and efficiently learn behaviors by backpropagating analytic gradients of learned state values through trajectories imagined in the compact state space of a learned world model.

Abstract

To select effective actions in complex environments, intelligent agents need to generalize from past experience. World models can represent knowledge about the environment to facilitate such generalization. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks purely by latent imagination. We efficiently learn behaviors by backpropagating analytic gradients of learned state values through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance.

Authors

Mohammad Norouzi

14 papers

T. Lillicrap

23 papers

Danijar Hafner

4 papers

References71 items

Mastering Atari, Go, chess and shogi by planning with a learned model

Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models

Benchmarking Model-Based Reinforcement Learning

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model

Shaping Belief States with Generative Environment Models for RL

Dream to Control: Learning Behaviors by Latent Imagination

Published in

International Conference on Learning Representa...(2019)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Mohammad Norouzi

14 papers

T. Lillicrap

23 papers

Danijar Hafner

4 papers

References71 items

Mastering Atari, Go, chess and shogi by planning with a learned model

Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models

Benchmarking Model-Based Reinforcement Learning

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model

Shaping Belief States with Generative Environment Models for RL

Exploring Model-based Planning with Policy Networks

DeepMDP: Learning Continuous Latent Space Models for Representation Learning

On Variational Bounds of Mutual Information

Model-Based Reinforcement Learning for Atari

PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos

Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic

Learning Latent Dynamics for Planning from Pixels

Formal Limitations on the Measurement of Mutual Information

Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Dopamine: A Research Framework for Deep Reinforcement Learning

Neural Predictive Belief Representations

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning

Representation Learning with Contrastive Predictive Coding

Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

Neural scene representation and rendering

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Learning Real-World Robot Policies by Dreaming

Universal Planning Networks

Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

Distributed Distributional Deterministic Policy Gradients

Model-Ensemble Trust-Region Policy Optimization

Learning and Querying Fast Generative Models for Reinforcement Learning

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Probabilistic Recurrent State-Space Models

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

DeepMind Control Suite

Mastering the game of Go without human knowledge

Self-Supervised Visual Planning with Temporal Skip Connections

Rainbow: Combining Improvements in Deep Reinforcement Learning

Robust Locally-Linear Controllable Embedding

Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

Proximal Policy Optimization Algorithms

Imagination-Augmented Agents for Deep Reinforcement Learning

Value Prediction Network

Model-Based Planning in Discrete Action Spaces

Model-Based Planning with Discrete and Continuous Actions

Reinforcement Learning with Unsupervised Auxiliary Tasks

Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data

Asynchronous Methods for Deep Reinforcement Learning

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Deep Kalman Filters

Learning Continuous Control Policies by Stochastic Value Gradients

Continuous control with deep reinforcement learning

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

Human-level control through deep reinforcement learning

Adam: A Method for Stochastic Optimization

Deterministic Policy Gradient Algorithms

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

The Arcade Learning Environment: An Evaluation Platform for General Agents

Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

The information bottleneck method

An Introduction to Variational Methods for Graphical Models

Dyna, an integrated architecture for learning, planning, and reacting

Backpropagation Applied to Handwritten Zip Code Recognition

2019), we fix it to 2 for all environments

Deep Variational Information Bottleneck

Tensorﬂow distributions

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

A New Approach to Linear Filtering and Prediction Problems

Reinforcement Learning: An Introduction

Making the world differentiable: on using self supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/1912.01603

Venue Information

Name

International Conference on Learning Representations

Type

conference

URL

https://iclr.cc/

Alternate Names

Int Conf Learn Represent
ICLR

TL;DR

Abstract

Authors

References71 items

Mastering Atari, Go, chess and shogi by planning with a learned model

Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models

Benchmarking Model-Based Reinforcement Learning

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model

Shaping Belief States with Generative Environment Models for RL

TL;DR

Abstract

Authors

References71 items

Mastering Atari, Go, chess and shogi by planning with a learned model

Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models

Benchmarking Model-Based Reinforcement Learning

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model

Shaping Belief States with Generative Environment Models for RL

Exploring Model-based Planning with Policy Networks

DeepMDP: Learning Continuous Latent Space Models for Representation Learning

On Variational Bounds of Mutual Information

Model-Based Reinforcement Learning for Atari

PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos

Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic

Learning Latent Dynamics for Planning from Pixels

Formal Limitations on the Measurement of Mutual Information

Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Dopamine: A Research Framework for Deep Reinforcement Learning

Neural Predictive Belief Representations

SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning

Representation Learning with Contrastive Predictive Coding

Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

Neural scene representation and rendering

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Learning Real-World Robot Policies by Dreaming

Universal Planning Networks

World Models

Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

Distributed Distributional Deterministic Policy Gradients

Model-Ensemble Trust-Region Policy Optimization

Learning and Querying Fast Generative Models for Reinforcement Learning

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Probabilistic Recurrent State-Space Models

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

DeepMind Control Suite

Mastering the game of Go without human knowledge

Self-Supervised Visual Planning with Temporal Skip Connections

Rainbow: Combining Improvements in Deep Reinforcement Learning

Robust Locally-Linear Controllable Embedding

Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

Proximal Policy Optimization Algorithms

Imagination-Augmented Agents for Deep Reinforcement Learning

Value Prediction Network

Model-Based Planning in Discrete Action Spaces

Model-Based Planning with Discrete and Continuous Actions

Reinforcement Learning with Unsupervised Auxiliary Tasks

Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data

Asynchronous Methods for Deep Reinforcement Learning

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Deep Kalman Filters

Learning Continuous Control Policies by Stochastic Value Gradients

Continuous control with deep reinforcement learning

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

Human-level control through deep reinforcement learning

Adam: A Method for Stochastic Optimization

Deterministic Policy Gradient Algorithms

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

The Arcade Learning Environment: An Evaluation Platform for General Agents

Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

The information bottleneck method

An Introduction to Variational Methods for Graphical Models

Dyna, an integrated architecture for learning, planning, and reacting

Backpropagation Applied to Handwritten Zip Code Recognition

2019), we fix it to 2 for all environments

each environment

Deep Variational Information Bottleneck

Tensorﬂow distributions

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning