Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations (2022-06-09T00:00:00.000000Z)

TL;DR

This paper introduces a suite of benchmarking tasks and shows that simple modifications to two popular vision-based online reinforcement learning algorithms, DreamerV2 and DrQ-v2, suffice to outperform existing offline RL methods and establish competitive baselines for continuous control in the visual domain.

Abstract

Offline reinforcement learning has shown great promise in leveraging large pre-collected datasets for policy learning, allowing agents to forgo often-expensive online data collection. However, offline reinforcement learning from visual observations with continuous action spaces remains under-explored, with a limited understanding of the key challenges in this complex domain. In this paper, we establish simple baselines for continuous control in the visual domain and introduce a suite of benchmarking tasks for offline reinforcement learning from visual observations designed to better represent the data distributions present in real-world offline RL problems and guided by a set of desiderata for offline RL from visual observations, including robustness to visual distractions and visually identifiable changes in dynamics. Using this suite of benchmarking tasks, we show that simple modifications to two popular vision-based online reinforcement learning algorithms, DreamerV2 and DrQ-v2, suffice to outperform existing offline RL methods and establish competitive baselines for continuous control in the visual domain. We rigorously evaluate these algorithms and perform an empirical evaluation of the differences between state-of-the-art model-based and model-free offline RL methods for continuous control from visual observations. All code and data used in this evaluation are open-sourced to facilitate progress in this domain.

Authors

Tim G. J. Rudner

5 papers

Cong Lu

1 papers

Philip J. Ball

1 papers

TL;DR

Abstract

Authors

References71 items

Learning Robust Dynamics through Variational Sparse Gating

In Defense of the Unitary Scalarization for Deep Multi-Task Learning

Offline Reinforcement Learning with Implicit Q-Learning

Revisiting Design Choices in Offline Model Based Reinforcement Learning

Implicit Behavioral Cloning

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning

Temporal Predictive Coding For Model-Based Planning In Latent Space

A Minimalist Approach to Offline Reinforcement Learning

Decision Transformer: Reinforcement Learning via Sequence Modeling

Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills

Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment

Decoupling Value and Policy for Generalization in Reinforcement Learning

COMBO: Conservative Offline Model-Based Policy Optimization

The Distracting Control Suite - A Challenging Benchmark for Reinforcement Learning from Pixels

Offline Reinforcement Learning from Images with Latent Space Models

ViNG: Learning Open-World Navigation with Visual Goals

Mastering Atari with Discrete World Models

Model-Based Offline Planning

Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction

Improving robustness against common corruptions by covariate shift adaptation

RL Unplugged: Benchmarks for Offline Reinforcement Learning

dm_control: Software and Tasks for Continuous Control

Array programming with NumPy

Conservative Q-Learning for Offline Reinforcement Learning

Language Models are Few-Shot Learners

MOPO: Model-based Offline Policy Optimization

Planning to Explore via Self-Supervised World Models

MOReL : Model-Based Offline Reinforcement Learning

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

Reinforcement Learning with Augmented Data

Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Observational Overfitting in Reinforcement Learning

Dream to Control: Learning Behaviors by Latent Imagination

Scaling data-driven robotics with reward sketching and batch reinforcement learning

Behavior Regularized Offline Reinforcement Learning

Reinforcement learning

An Optimistic Perspective on Offline Reinforcement Learning

Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables

To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

Soft Actor-Critic Algorithms and Applications

Off-Policy Deep Reinforcement Learning without Exploration

Disentangling Disentanglement in Variational Autoencoders

Learning Latent Dynamics for Planning from Pixels

Pre-training on Grayscale ImageNet Improves Medical Image Classification

Learning to Drive in a Day

QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning

Understanding disentangling in β-VAE

Addressing Function Approximation Error in Actor-Critic Methods

State Representation Learning for Control: An Overview

Robust and Efficient Transfer Learning with Hidden Parameter Markov Decision Processes

1 year, 1000 km: The Oxford RobotCar dataset

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Continuous control with deep reinforcement learning

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

ImageNet classification with deep convolutional neural networks

Tree-Based Batch Mode Reinforcement Learning

Behavioural Cloning: Phenomena, Results and Problems

Should I Run Offline Reinforcement Learning or Behavioral Cloning?

exploitation when training solely on the offline data, even with an uncertainty penalty

VariBAD: Variational Bayes-Adaptive Deep RL via Meta-Learning

Learning Robust State Abstractions for Hidden-Parameter Block MDPs

Following the notation from Yarats et al. (2021a), the DrQ-v2 actor πφ is trained with the following loss

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Figure 3, we note that settings with a mixture of distractors often had worse test-time transfer performance than those with a single distractor

A Framework for Behavioural Cloning

Estimating the mean and variance of the target probability distribution

MTEnv -environment interface for mulit-task reinforcement learning. Github, 2021

Field of Study

Journal Information

Name

Volume