Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose? (2021-07-24T00:00:00.000000Z)

TL;DR

It is found that on an environment that requires multimodal posterior predictives, mixture density nets outperform all other models by a large margin, and that heteroscedasticity at training time, perhaps acting as a regularizer, improves predictions at longer horizons.

Abstract

We contribute to micro-data model-based reinforcement learning (MBRL) by rigorously comparing popular generative models using a fixed (random shooting) control agent. We find that on an environment that requires multimodal posterior predictives, mixture density nets outperform all other models by a large margin. When multimodality is not required, our surprising finding is that we do not need probabilistic posterior predictives: deterministic models are on par, in fact they consistently (although non-significantly) outperform their probabilistic counterparts. We also found that heteroscedasticity at training time, perhaps acting as a regularizer, improves predictions at longer horizons. At the methodological side, we design metrics and an experimental protocol which can be used to evaluate the various models, predicting their asymptotic performance when using them on the control problem. Using this framework, we improve the state-of-the-art sample complexity of MBRL on Acrobot by two to four folds, using an aggressive training schedule which is outside of the hyperparameter interval usually considered

Authors

B. Kégl

1 papers

Gabriel Hurtado

1 papers

Albert Thomas

1 papers

TL;DR

Abstract

Authors

References56 items

On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning

Model-Based Offline Planning

Selective Dyna-style Planning Under Limited Model Capacity

Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Agent57: Outperforming the Atari Human Benchmark

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Grandmaster level in StarCraft II using multi-agent reinforcement learning

Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods

Benchmarking Model-Based Reinforcement Learning

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model

Exploring Model-based Planning with Policy Networks

Calibrated Model-Based Deep Reinforcement Learning

Ray Interference: a Source of Plateaus in Deep Reinforcement Learning

Supervised Anomaly Detection based on Deep Autoregressive Density Estimators

A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play

GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration

Modeling the Long Term Future in Model-Based Reinforcement Learning

Recurrent World Models Facilitate Policy Evolution

A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials

Randomized Prior Functions for Deep Reinforcement Learning

The RAMP framework: from reproducibility to transparency in the design and optimization of scientific workflows

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning

Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

An autoregressive recurrent mixture density network for parametric speech synthesis

Masked Autoregressive Flow for Density Estimation

Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning

Micro-Data Learning: The Other End of the Spectrum

OpenAI Gym

Density estimation using Real NVP

Learning Structured Output Representation using Deep Conditional Generative Models

Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission

Variational Inference with Normalizing Flows

Modelling acoustic feature dependencies with artificial neural networks: Trajectory-RNADE

Gaussian Processes for Data-Efficient Learning in Robotics and Control

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

RNADE: The real-valued neural autoregressive density-estimator

Learning Stable Nonlinear Dynamical Systems With Gaussian Mixture Models

PILCO: A Model-Based and Data-Efficient Approach to Policy Search

Scikit-learn: Machine Learning in Python

Dynamic programming for global control of the acrobot and its chaotic aspect

Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding

AUTO-ENCODING VARIATIONAL BAYES

U NCERTAINTY W EIGHTED O FFLINE R EINFORCEMENT L EARNING

R4) Heteroscedastic models are able to vary their uncertainty estimate as a function of the state or trace

Uncertainty in Deep Learning

Improving PILCO with Bayesian Neural Network Dynamics Models

2017): it is the number of coupling layers. • CVAE (Sohn et al., 2015): it is the total number of layers (encoder plus decoder)

R7) Complex multivariate density estimators rarely work out of the box on a new system. We are aiming at reusability of our models (not simple reproducibility of our experimental results)

2015): it is the total number of layers (encoder plus decoder). ”Nb components” is the number of components in the outputted density mixture. In the GP and deterministic NN

( Preprint ) AAS 09-334 A SURVEY OF NUMERICAL METHODS FOR OPTIMAL CONTROL

A Tutorial on the Cross-Entropy Method

Robust constrained model predictive control

Mixture Density Networks

R5) Allowing multi-modal posterior predictives seems to be crucial to properly handle uncertainty around discrete jumps in the system state that lead to qualitatively different futures

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names