Dynamics-Aware … (2019-07-02T00:00:00.000000Z)

TL;DR

This work proposes an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics, and demonstrates that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, and substantially improves over prior hierarchical RL methods for unsuper supervised skill discovery.

Abstract

Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks. However, learning an accurate model for complex dynamical systems is difficult, and even then, the model might not generalize well outside the distribution of states on which it was trained. In this work, we combine model-based learning with model-free learning of primitives that make model-based planning easy. To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict? We propose an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics. Our method can leverage continuous skill spaces, theoretically, allowing us to learn infinitely many behaviors even for high-dimensional state-spaces. We demonstrate that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, can handle sparse-reward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery.

Authors

S. Levine

59 papers

Vikash Kumar

7 papers

S. Gu

11 papers

References77 items

Soft Actor-Critic Algorithms and Applications

Unsupervised Control Through Non-Parametric Discriminative Rewards

TherML: The Thermodynamics of Machine Learning

Recurrent World Models Facilitate Policy Evolution

Variational Option Discovery Algorithms

Abstract

TL;DR

Abstract

Authors

References77 items

Soft Actor-Critic Algorithms and Applications

Unsupervised Control Through Non-Parametric Discriminative Rewards

TherML: The Thermodynamics of Machine Learning

Recurrent World Models Facilitate Policy Evolution

Variational Option Discovery Algorithms

TL;DR

Abstract

Authors

References77 items

Soft Actor-Critic Algorithms and Applications

Unsupervised Control Through Non-Parametric Discriminative Rewards

TherML: The Thermodynamics of Machine Learning

Recurrent World Models Facilitate Policy Evolution

Variational Option Discovery Algorithms

TherML: Thermodynamics of Machine Learning

QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Data-Efficient Hierarchical Reinforcement Learning

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review

Disentangling the independently controllable factors of variation by interacting with the world

Diversity is All You Need: Learning Skills without a Reward Function

Model-Ensemble Trust-Region Policy Optimization

Learning an Embedding Space for Transferable Robot Skills

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

DeepLoco

Proximal Policy Optimization Algorithms

Emergence of Locomotion Behaviours in Rich Environments

Hindsight Experience Replay

Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control

Curiosity-Driven Exploration by Self-Supervised Prediction

Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning

FeUdal Networks for Hierarchical Reinforcement Learning

EX2: Exploration with Exemplar Models for Deep Reinforcement Learning

Variational Intrinsic Control

#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

Stochastic Neural Networks for Hierarchical Reinforcement Learning

Learning and Transfer of Modulated Locomotor Controllers

Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates

The Option-Critic Architecture

Unifying Count-Based Exploration and Intrinsic Motivation

OpenAI Gym

Curiosity-driven Exploration in Deep Reinforcement Learning via Bayesian Neural Networks

Aggressive driving with model predictive path integral control

Optimal control with learned local models: Application to dexterous manipulation

Mastering the game of Go with deep neural networks and tree search

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

One-shot learning of manipulation skills with online dynamics adaptation and neural network priors

Action-Conditional Video Prediction using Deep Networks in Atari Games

DeepMPC: Learning Deep Latent Features for Model Predictive Control

Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

End-to-End Training of Deep Visuomotor Policies

Trust Region Policy Optimization

Gaussian Processes for Data-Efficient Learning in Robotics and Control

Adam: A Method for Stochastic Optimization

Hierarchical Relative Entropy Policy Search

Playing Atari with Deep Reinforcement Learning

MuJoCo: A physics engine for model-based control

PILCO: A Model-Based and Data-Efficient Approach to Policy Search

Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010)

Maximum Entropy Inverse Reinforcement Learning

What is Intrinsic Motivation? A Typology of Computational Approaches

Gaussian Processes and Reinforcement Learning for Identification and Control of an Autonomous Blimp

Empowerment: a universal agent-centric measure of control

Estimating mutual information and multi-information in large networks

Gaussian process model based predictive control

Gaussian Processes For Machine Learning

The IM algorithm: a variational approach to Information Maximization

Learning Options in Reinforcement Learning

Multivariate Information Bottleneck

The information bottleneck method

Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

Using Options for Knowledge Transfer in Reinforcement Learning

Adaptive Mixtures of Local Experts