Decision Transformer: Reinforcement Learning via Sequence Modeling (2021-06-02T00:00:00.000000Z)

TL;DR

Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

Abstract

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

Authors

Aditya Grover

4 papers

P. Abbeel

35 papers

A. Srinivas

6 papers

TL;DR

Abstract

Authors

References91 items

Pretrained Transformers as Universal Computation Engines

Zero-Shot Text-to-Image Generation

Synthetic Returns for Long-Term Credit Assignment

Imitating Interactive Intelligence

Reset-Free Lifelong Learning with Skill-Space Planning

Planning from Pixels using Inverse Dynamics Models

Parrot: Data-Driven Behavioral Priors for Reinforcement Learning

Counterfactual Credit Assignment in Model-Free Reinforcement Learning

Transformers for One-Shot Visual Imitation

OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Accelerating Reinforcement Learning with Learned Skill Priors

Mastering Atari with Discrete World Models

Transformers: State-of-the-Art Natural Language Processing

GeDi: Generative Discriminator Guided Sequence Generation

An Optimistic Perspective on Offline Deep Reinforcement Learning

Generative Pretraining From Pixels

Accelerating Online Reinforcement Learning with Offline Datasets

Conservative Q-Learning for Offline Reinforcement Learning

Rapid Task-Solving in Novel Environments

Language Models are Few-Shot Learners

MOPO: Model-based Offline Policy Optimization

End-to-End Object Detection with Transformers

MOReL : Model-Based Offline Reinforcement Learning

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills

Reward-Conditioned Policies

Learning to Reach Goals via Iterated Supervised Learning

Training Agents using Upside-Down Reinforcement Learning

Hindsight Credit Assignment

Stabilizing Transformers for Reinforcement Learning

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

Plug and Play Language Models: A Simple Approach to Controlled Text Generation

Behavior Regularized Offline Reinforcement Learning

Learning To Reach Goals Without Reinforcement Learning

Fine-Tuning Language Models from Human Preferences

CTRL: A Conditional Transformer Language Model for Controllable Generation

Self-Attentional Credit Assignment for Transfer in Reinforcement Learning

An Optimistic Perspective on Offline Reinforcement Learning

Dynamics-Aware Unsupervised Discovery of Skills

When to Trust Your Model: Model-Based Policy Optimization

Explain Yourself! Leveraging Language Models for Commonsense Reasoning

Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning

Go-Explore: a New Approach for Hard-Exploration Problems

A Style-Based Generator Architecture for Generative Adversarial Networks

Off-Policy Deep Reinforcement Learning without Exploration

Optimizing agent behavior over long time scales by transporting value

Deep reinforcement learning with relational inductive biases

RUDDER: Return Decomposition for Delayed Rewards

Learning to Write with Cooperative Discriminators

Diversity is All You Need: Learning Skills without a Reward Function

Reinforcement Learning from Imperfect Demonstrations

Decoupled Weight Decay Regularization

Distributional Reinforcement Learning with Quantile Regression

Controlling Linguistic Style Aspects in Neural Language Generation

Hafez: an Interactive Poetry Generation System

Attention is All you Need

Toward Controlled Generation of Text

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

Layer Normalization

Human-level control through deep reinforcement learning

Playing Atari with Deep Reinforcement Learning

The Arcade Learning Environment: An Evaluation Platform for General Agents

Learning from delayed rewards

Reinforcement Learning as One Big Sequence Modeling Problem

Controllable neural text generation. lilianweng.github.io/lillog

BC scores, we use the same hyperparameters as Decision Transformer for fair comparison. For REM and QR-DQN, there is a slight discrepancy between Agarwal et al

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

URL https://ogma

Acting without rewards

Improving Language Understanding by Generative Pre-Training

Reinforcement Learning: An Introduction