When to Trust Y… (2019-06-19T00:00:00.000000Z)

TL;DR

This paper first formulate and analyze a model-based reinforcement learning algorithm with a guarantee of monotonic improvement at each step, and demonstrates that a simple procedure of using short model-generated rollouts branched from real data has the benefits of more complicated model- based algorithms without the usual pitfalls.

Abstract

Designing effective model-based reinforcement learning algorithms is difficult because the ease of data generation must be weighed against the bias of model-generated data. In this paper, we study the role of model usage in policy optimization both theoretically and empirically. We first formulate and analyze a model-based reinforcement learning algorithm with a guarantee of monotonic improvement at each step. In practice, this analysis is overly pessimistic and suggests that real off-policy data is always preferable to model-generated on-policy data, but we show that an empirical estimate of model generalization can be incorporated into such analysis to justify model usage. Motivated by this analysis, we then demonstrate that a simple procedure of using short model-generated rollouts branched from real data has the benefits of more complicated model-based algorithms without the usual pitfalls. In particular, this approach surpasses the sample efficiency of prior model-based methods, matches the asymptotic performance of the best model-free algorithms, and scales to horizons that cause other model-based methods to fail entirely.

Authors

S. Levine

59 papers

Justin Fu

5 papers

Michael Janner

1 papers

References45 items

Task-Agnostic Dynamics Priors for Deep Reinforcement Learning

Model-Based Reinforcement Learning for Atari

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

Probabilistic Planning with Sequential Monte Carlo methods

Understanding the Asymptotic Performance of Model-Based RL Methods

Abstract

References45 items

TL;DR

Abstract

Authors

References45 items

Task-Agnostic Dynamics Priors for Deep Reinforcement Learning

Model-Based Reinforcement Learning for Atari

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

Probabilistic Planning with Sequential Monte Carlo methods

Understanding the Asymptotic Performance of Model-Based RL Methods

TL;DR

Abstract

Authors

References45 items

Task-Agnostic Dynamics Priors for Deep Reinforcement Learning

Model-Based Reinforcement Learning for Atari

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

Probabilistic Planning with Sequential Monte Carlo methods

Understanding the Asymptotic Performance of Model-Based RL Methods

Model-Based Reinforcement Learning via Meta-Policy Optimization

Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

The Effect of Planning Shape on Dyna-style Planning in High-dimensional State Spaces

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Dual Policy Iteration

Lipschitz Continuity in Model-based Reinforcement Learning

Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

Model-Ensemble Trust-Region Policy Optimization

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Uncertainty-driven Imagination for Continuous Deep Reinforcement Learning

On the Sample Complexity of the Linear Quadratic Regulator

Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

Proximal Policy Optimization Algorithms

Imagination-Augmented Agents for Deep Reinforcement Learning

Value Prediction Network

Value-Aware Loss Function for Model-based Reinforcement Learning

The Predictron: End-To-End Learning and Planning

Self-Correcting Models for Model-Based Reinforcement Learning

EPOpt: Learning Robust Neural Network Policies Using Model Ensembles

Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks

Optimal control with learned local models: Application to dexterous manipulation

Continuous Deep Q-Learning with Model-based Acceleration

Value Iteration Networks

Learning Continuous Control Policies by Stochastic Value Gradients

Continuous control with deep reinforcement learning

Action-Conditional Video Prediction using Deep Networks in Atari Games

Human-level control through deep reinforcement learning

Trust Region Policy Optimization

Model Regularization for Stable Sample Rollouts

Guided Policy Search

MuJoCo: A physics engine for model-based control

PILCO: A Model-Based and Data-Efficient Approach to Policy Search

Model-based reinforcement learning with nearly tight exploration complexity bounds

Learning tasks from a single demonstration

Reinforcement Learning: A Survey

Model Predictive Control Using Neural Networks [25 Years Ago]

Improving PILCO with Bayesian Neural Network Dynamics Models

Understanding Machine Learning - From Theory to Algorithms

Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names