Learning State Representations from Random Deep Action-conditional Predictions (2021-02-09T00:00:00.000000Z)

TL;DR

This work shows that random deep action-conditional predictions when used as auxiliary tasks yield state representations that produce control performance competitive with state-of-the-art hand-crafted auxiliary tasks like value prediction, pixel control, and CURL in both Atari and DeepMind Lab tasks.

Abstract

Our main contribution in this work is an empirical finding that random General Value Functions (GVFs), i.e., deep action-conditional predictions -- random both in what feature of observations they predict as well as in the sequence of actions the predictions are conditioned upon -- form good auxiliary tasks for reinforcement learning (RL) problems. In particular, we show that random deep action-conditional predictions when used as auxiliary tasks yield state representations that produce control performance competitive with state-of-the-art hand-crafted auxiliary tasks like value prediction, pixel control, and CURL in both Atari and DeepMind Lab tasks. In another set of experiments we stop the gradients from the RL part of the network to the state representation learning part of the network and show, perhaps surprisingly, that the auxiliary tasks alone are sufficient to learn state representations good enough to outperform an end-to-end trained actor-critic baseline. We opensourced our code at https://github.com/Hwhitetooth/random_gvfs.

Authors

Satinder Singh

3 papers

Zeyu Zheng

1 papers

Vivek Veeriah

1 papers

TL;DR

Abstract

Authors

References41 items

On The Effect of Auxiliary Tasks on Representation Dynamics

Mastering Atari with Discrete World Models

Decoupling Representation Learning from Reinforcement Learning

Predictive Information Accelerates Learning in RL

Data-Efficient Reinforcement Learning with Self-Predictive Representations

Deep Reinforcement and InfoMax Learning

The Value-Improvement Path: Towards Better Representations for Reinforcement Learning

Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning

CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Discovery of Useful Questions as Auxiliary Tasks

Terminal Prediction as an Auxiliary Task for Deep Reinforcement Learning

Shaping Belief States with Generative Environment Models for RL

Unsupervised State Representation Learning in Atari

Hyperbolic Discounting and Learning over Multiple Horizons

A Geometric Perspective on Optimal Representations for Reinforcement Learning

Neural Predictive Belief Representations

General Value Function Networks

Representation Learning with Contrastive Predictive Coding

Reinforcement Learning with Unsupervised Auxiliary Tasks

Asynchronous Methods for Deep Reinforcement Learning

Dueling Network Architectures for Deep Reinforcement Learning

Human-level control through deep reinforcement learning

Adam: A Method for Stochastic Optimization

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

The Arcade Learning Environment: An Evaluation Platform for General Agents

Horde: a scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction

Temporal Abstraction in Temporal-difference Networks

Temporal-Difference Networks

Predictive State Representations: A New Theory for Modeling Dynamical Systems

Predictive Representations of State

with respect to the random seed after running experiments multiple times)? [Yes] See Figure 3b

(a) Did you state the full set of assumptions of all theoretical results

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation

Did you discuss whether and how consent was obtained from people whose data you're using/curating?

Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix

Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Have you read the ethics review guidelines and ensured that your paper conforms to them

code, data, models) or curating/releasing new assets... (a) If your work uses existing assets

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

(a) Did you include the code, data, and instructions needed to reproduce the main experimental results

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names