Decision Transformer: Reinforcement Learning via Sequence Modeling

Published in

Neural Information Processing Systems(2021)

External Links:

Generate Graph

TL;DR

Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

Abstract

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

Authors

Aditya Grover

4 papers

P. Abbeel

35 papers

A. Srinivas

6 papers

References91 items

Pretrained Transformers as Universal Computation Engines

Zero-Shot Text-to-Image Generation

Synthetic Returns for Long-Term Credit Assignment

Imitating Interactive Intelligence

Reset-Free Lifelong Learning with Skill-Space Planning

Decision Transformer: Reinforcement Learning via Sequence Modeling

Published in

Neural Information Processing Systems(2021)

External Links:

Generate Graph

TL;DR

Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

Abstract

Authors

Aditya Grover

4 papers

P. Abbeel

35 papers

A. Srinivas

6 papers

References91 items

Pretrained Transformers as Universal Computation Engines

Zero-Shot Text-to-Image Generation

Synthetic Returns for Long-Term Credit Assignment

Imitating Interactive Intelligence

Reset-Free Lifelong Learning with Skill-Space Planning

A. Rajeswaran

6 papers

Igor Mordatch

6 papers

Planning from Pixels using Inverse Dynamics Models

Parrot: Data-Driven Behavioral Priors for Reinforcement Learning

Counterfactual Credit Assignment in Model-Free Reinforcement Learning

Transformers for One-Shot Visual Imitation

OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Accelerating Reinforcement Learning with Learned Skill Priors

Mastering Atari with Discrete World Models

Transformers: State-of-the-Art Natural Language Processing

GeDi: Generative Discriminator Guided Sequence Generation

An Optimistic Perspective on Offline Deep Reinforcement Learning

Generative Pretraining From Pixels

Accelerating Online Reinforcement Learning with Offline Datasets

Conservative Q-Learning for Offline Reinforcement Learning

Rapid Task-Solving in Novel Environments

Language Models are Few-Shot Learners

MOPO: Model-based Offline Policy Optimization

End-to-End Object Detection with Transformers

MOReL : Model-Based Offline Reinforcement Learning

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills

Reward-Conditioned Policies

Learning to Reach Goals via Iterated Supervised Learning

Training Agents using Upside-Down Reinforcement Learning

Hindsight Credit Assignment

Stabilizing Transformers for Reinforcement Learning

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

Plug and Play Language Models: A Simple Approach to Controlled Text Generation

Behavior Regularized Offline Reinforcement Learning

Learning To Reach Goals Without Reinforcement Learning

Fine-Tuning Language Models from Human Preferences

CTRL: A Conditional Transformer Language Model for Controllable Generation

Self-Attentional Credit Assignment for Transfer in Reinforcement Learning

An Optimistic Perspective on Offline Reinforcement Learning

Dynamics-Aware Unsupervised Discovery of Skills

When to Trust Your Model: Model-Based Policy Optimization

Explain Yourself! Leveraging Language Models for Commonsense Reasoning

Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning

Go-Explore: a New Approach for Hard-Exploration Problems

A Style-Based Generator Architecture for Generative Adversarial Networks

Off-Policy Deep Reinforcement Learning without Exploration

Optimizing agent behavior over long time scales by transporting value

Deep reinforcement learning with relational inductive biases

RUDDER: Return Decomposition for Delayed Rewards

Learning to Write with Cooperative Discriminators

Diversity is All You Need: Learning Skills without a Reward Function

Reinforcement Learning from Imperfect Demonstrations

Decoupled Weight Decay Regularization

Distributional Reinforcement Learning with Quantile Regression

Controlling Linguistic Style Aspects in Neural Language Generation

Hafez: an Interactive Poetry Generation System

Attention is All you Need

Toward Controlled Generation of Text

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

Layer Normalization

Human-level control through deep reinforcement learning

Playing Atari with Deep Reinforcement Learning

The Arcade Learning Environment: An Evaluation Platform for General Agents

Learning from delayed rewards

Reinforcement Learning as One Big Sequence Modeling Problem

Controllable neural text generation. lilianweng.github.io/lillog

BC scores, we use the same hyperparameters as Decision Transformer for fair comparison. For REM and QR-DQN, there is a slight discrepancy between Agarwal et al

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Acting without rewards

Improving Language Understanding by Generative Pre-Training

Reinforcement Learning: An Introduction

Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming

Summary: pretrained language models can exhibit cross-modal transfer to non-language modalities

Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes

Did you discuss any potential negative societal impacts of your work?

(a) Did you state the full set of assumptions of all theoretical results

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation

Did you discuss whether and how consent was obtained from people whose data you're using/curating?

Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Have you read the ethics review guidelines and ensured that your paper conforms to them

code, data, models) or curating/releasing new assets... (a) If your work uses existing assets

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots

Did you describe the limitations of your work? [Yes] Mainly discussed in conclusion, but also discussed in Section 5

(a) Did you include the code, data, and instructions needed to reproduce the main experimental results

Medium-Replay: the replay buffer of an agent trained to the performance of a medium policy (approximately 25k-400k timesteps in our environments)

Field of Study

Computer Science

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS