COMBO: Conservative Offline Model-Based Policy Optimization (2021-02-16T00:00:00.000000Z)

TL;DR

This work develops a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model, and finds that it consistently performs as well or better as compared to prior offline model-free and model- based methods on widely studied offline RL benchmarks, including image-based tasks.

Abstract

Model-based algorithms, which learn a dynamics model from logged experience and perform some sort of pessimistic planning under the learned model, have emerged as a promising paradigm for offline reinforcement learning (offline RL). However, practical variants of such model-based algorithms rely on explicit uncertainty quantification for incorporating pessimism. Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model. This results in a conservative estimate of the value function for out-of-support state-action tuples, without requiring explicit uncertainty estimation. We theoretically show that our method optimizes a lower bound on the true policy value, that this bound is tighter than that of prior methods, and our approach satisfies a policy improvement guarantee in the offline setting. Through experiments, we find that COMBO consistently performs as well or better as compared to prior offline model-free and model-based methods on widely studied offline RL benchmarks, including image-based tasks.

Authors

Chelsea Finn

26 papers

S. Levine

59 papers

Aviral Kumar

6 papers

TL;DR

Abstract

Authors

References74 items

A Minimalist Approach to Offline Reinforcement Learning

Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning

Model-Based Offline Planning with Trajectory Pruning

Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism

Offline Reinforcement Learning with Fisher Divergence Critic Regularization

S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning

Is Pessimism Provably Efficient for Offline RL?

Offline Reinforcement Learning from Images with Latent Space Models

PLAS: Latent Action Space for Offline Reinforcement Learning

COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning

Human-centric Dialog Training via Offline Reinforcement Learning

Overcoming Model Bias for Robust Offline Deep Reinforcement Learning

Model-Based Offline Planning

EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL

Provably Good Batch Reinforcement Learning Without Great Exploration

Conservative Q-Learning for Offline Reinforcement Learning

Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization

MOPO: Model-based Offline Policy Optimization

MOReL : Model-Based Offline Reinforcement Learning

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Model-Augmented Actor-Critic: Backpropagating through Paths

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning

AlgaeDICE: Policy Gradient from Arbitrary Experience

IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

Behavior Regularized Offline Reinforcement Learning

An Optimistic Perspective on Offline Reinforcement Learning

Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog

When to Trust Your Model: Model-Based Policy Optimization

Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

Information-Theoretic Considerations in Batch Reinforcement Learning

Off-Policy Deep Reinforcement Learning without Exploration

Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigation

Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation

QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

Randomized Prior Functions for Deep Reinforcement Learning

BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning

Addressing Function Approximation Error in Actor-Critic Methods

Efficient Exploration Through Bayesian Deep Q-Networks

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

DeepMind Control Suite

Safe Policy Improvement with Baseline Bootstrapping

Deep visual foresight for planning robot motion

Why is Posterior Sampling Better than Optimism for Reinforcement Learning?

Safe Policy Improvement by Minimizing Robust Baseline Regret

An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning

Agnostic System Identification for Model-Based Reinforcement Learning

Informing sequential clinical decision-making through reinforcement learning: an empirical study

Near-optimal Regret Bounds for Reinforcement Learning

Finite-Time Bounds for Fitted Value Iteration

Tree-Based Batch Mode Reinforcement Learning

Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method

Off-Policy Temporal Difference Learning with Function Approximation

Dyna, an integrated architecture for learning, planning, and reacting

Representation Balancing Offline Model-based Reinforcement Learning

We use the standard walker-walk environment

β, deviating away from the behavior policy is expected to lead to smaller values

multiplied by a fraction f ) when a near-accurate model is used to augment data for training the Q-function, and in contrast to prior model-based offline RL results Kidambi et al

Reinforcement Learning: Theory and Algorithms

International conference on machine learning

so, we formally define the standard concentration properties of the reward and transition dynamics in the empirical MDP,M, that we assume so as to prove Theorem 1

Batch learning from logged bandit feedback through counterfactual risk minimization

Batch Reinforcement Learning

Neuro-Dynamic Programming

Off‐Policy Actor‐Criticアルゴリズムによる強化学習

Reinforcement Learning: An Introduction

Bertsekas and John N . Tsitsiklis

Off-policy policy gradient with state distribution correction

Field of Study

Venue Information