A Minimalist Ap… (2021-06-12T00:00:00.000000Z)

TL;DR

This paper finds that it can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data.

Abstract

Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overhead of previous methods.

Authors

Scott Fujimoto

4 papers

S. Gu

11 papers

References64 items

Decision Transformer: Reinforcement Learning via Sequence Modeling

Benchmarks for Deep Off-Policy Evaluation

Offline Reinforcement Learning with Fisher Divergence Critic Regularization

Offline Policy Selection under Uncertainty

Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models

TL;DR

Abstract

Authors

Scott Fujimoto

4 papers

S. Gu

11 papers

TL;DR

Abstract

Authors

References64 items

Decision Transformer: Reinforcement Learning via Sequence Modeling

Benchmarks for Deep Off-Policy Evaluation

Offline Reinforcement Learning with Fisher Divergence Critic Regularization

Offline Policy Selection under Uncertainty

Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models

TL;DR

Abstract

Authors

References64 items

Decision Transformer: Reinforcement Learning via Sequence Modeling

Benchmarks for Deep Off-Policy Evaluation

Offline Reinforcement Learning with Fisher Divergence Critic Regularization

Offline Policy Selection under Uncertainty

Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models

The Importance of Pessimism in Fixed-Dataset Policy Optimization

EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL

Hyperparameter Selection for Offline Reinforcement Learning

Critic Regularized Regression

Accelerating Online Reinforcement Learning with Offline Datasets

Conservative Q-Learning for Offline Reinforcement Learning

Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Implementation Matters in Deep RL: A Case Study on PPO and TRPO

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning

AlgaeDICE: Policy Gradient from Arbitrary Experience

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Sparse Reward Environments

Benchmarking Batch Deep Reinforcement Learning Algorithms

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning

Behavior Regularized Offline Reinforcement Learning

Making Efficient Use of Demonstrations to Solve Hard Exploration Problems

Behaviour Suite for Reinforcement Learning

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog

Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

Off-Policy Deep Reinforcement Learning without Exploration

Exponentially Weighted Imitation Learning for Batched Historical Data

Horizon: Facebook's Open Source Applied Reinforcement Learning Platform

Policy Optimization with Demonstrations

Evaluating Reinforcement Learning Algorithms in Observational Health Settings

Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Mapless Navigation by Leveraging Prior Demonstrations

Addressing Function Approximation Error in Actor-Critic Methods

Reinforcement and Imitation Learning for Diverse Visuomotor Skills

The Mirage of Action-Dependent Baselines in Reinforcement Learning

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Safe Policy Improvement with Baseline Bootstrapping

Overcoming Exploration in Reinforcement Learning with Demonstrations

Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

Deep Reinforcement Learning that Matters

Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

Proximal Policy Optimization Algorithms

Deep Q-learning From Demonstrations

OpenAI Gym

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Continuous control with deep reinforcement learning

Adam: A Method for Stochastic Optimization

Deterministic Policy Gradient Algorithms

Imitation Learning with Demonstrations and Shaping Rewards

Offline policy evaluation across representations with applications to educational games

Learning from Limited Demonstrations

MuJoCo: A physics engine for model-based control

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Robot learning from demonstration

A Natural Policy Gradient

Efficient Training of Artificial Neural Networks for Autonomous Navigation

Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning

Batch Reinforcement Learning Through Continuation Method

What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study

BC + RL : Imitation Learning From Non-Optimal Demonstrations

Critic Learning Rate 3e-4 Actor Learning Rate 3e-4 Mini-batch size

Batch Reinforcement Learning

Reinforcement Learning: An Introduction

Field of Study

Journal Information

Name