Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks (2018-10-06T00:00:00.000000Z)

TL;DR

This work proposes to use convolutional network architectures to generate Q-values and updates for a large number of goals at once and shows that replacing the random actions in ε-greedy exploration by several actions towards feasible goals generates better exploratory trajectories on Montezuma's Revenge and Super Mario All-Stars games.

Abstract

Being able to reach any desired location in the environment can be a valuable asset for an agent. Learning a policy to navigate between all pairs of states individually is often not feasible. An all-goals updating algorithm uses each transition to learn Q-values towards all goals simultaneously and off-policy. However the expensive numerous updates in parallel limited the approach to small tabular cases so far. To tackle this problem we propose to use convolutional network architectures to generate Q-values and updates for a large number of goals at once. We demonstrate the accuracy and generalization qualities of the proposed method on randomly generated mazes and Sokoban puzzles. In the case of on-screen goal coordinates the resulting mapping from frames to distance-maps directly informs the agent about which places are reachable and in how many steps. As an example of application we show that replacing the random actions in ε-greedy exploration by several actions towards feasible goals generates better exploratory trajectories on Montezuma's Revenge and Super Mario All-Stars games.

Authors

Fabio Pardo

1 papers

Vitaly Levdik

1 papers

Petar Kormushev

1 papers

TL;DR

Abstract

Authors

References48 items

Compression and Localization in Reinforcement Learning for ATARI Games

Visual Reinforcement Learning with Imagined Goals

Many-Goals Reinforcement Learning

Disentangling Controllable and Uncontrollable Factors of Variation by Interacting with the World

Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

Time Limits in Reinforcement Learning

Action Branching Architectures for Deep Reinforcement Learning

Imagination-Augmented Agents for Deep Reinforcement Learning

Reverse Curriculum Generation for Reinforcement Learning

The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously

Hindsight Experience Replay

GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms

Hybrid Reward Architecture for Reinforcement Learning

Curiosity-Driven Exploration by Self-Supervised Prediction

FeUdal Networks for Hierarchical Reinforcement Learning

#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

Reinforcement Learning with Unsupervised Auxiliary Tasks

Unifying Count-Based Exploration and Intrinsic Motivation

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Deep Exploration via Bootstrapped DQN

Value Iteration Networks

Dueling Network Architectures for Deep Reinforcement Learning

Prioritized Experience Replay

Continuous control with deep reinforcement learning

Universal Value Function Approximators

Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Human-level control through deep reinforcement learning

Adam: A Method for Stochastic Optimization

Active learning of inverse models with intrinsically motivated goal exploration in robots

Horde: a scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction

Double Q-learning

Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010)

What is Intrinsic Motivation? A Typology of Computational Approaches

Efficient Reinforcement Learning in Factored MDPs

Q-learning

A possibility for implementing curiosity and boredom in model-building neural controllers

AUTO-ENCODING VARIATIONAL BAYES

OpenAI Retro

Gym-Sokoban

Hierarchical Reinforcement Learning Based on Subgoal Discovery and Subpolicy Specialization

and Schmidhuber

and Tennenholtz

R-MAX - A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning

and Koller

Reinforcement Learning: An Introduction

Learning to Achieve Goals

and Dayan

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION 1 Intrinsic Motivation Systems for Autonomous Mental Development

Field of Study

Venue Information

Name

Type

URL

Alternate Names