Asynchronous Methods for Deep Reinforcement Learning

Published in

International Conference on Machine Learning(2016)

External Links:

Generate Graph

TL;DR

A conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers and shows that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.

Abstract

We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.

Authors

K. Kavukcuoglu

26 papers

David Silver

21 papers

Volodymyr Mnih

6 papers

References43 items

Increasing the Action Gap: New Operators for Reinforcement Learning

True Online Temporal-Difference Learning

Dueling Network Architectures for Deep Reinforcement Learning

Prioritized Experience Replay

Deep Reinforcement Learning with Double Q-Learning

Asynchronous Methods for Deep Reinforcement Learning

Published in

International Conference on Machine Learning(2016)

External Links:

Generate Graph

TL;DR

Abstract

Authors

K. Kavukcuoglu

26 papers

David Silver

21 papers

Volodymyr Mnih

6 papers

References43 items

Increasing the Action Gap: New Operators for Reinforcement Learning

True Online Temporal-Difference Learning

Dueling Network Architectures for Deep Reinforcement Learning

Prioritized Experience Replay

Deep Reinforcement Learning with Double Q-Learning

Adrià Puigdomènech Badia

3 papers

Mehdi Mirza

6 papers

Alex Graves

11 papers

T. Lillicrap

23 papers

Continuous control with deep reinforcement learning

Distributed Deep Q-Learning

Massively Parallel Methods for Deep Reinforcement Learning

High-Dimensional Continuous Control Using Generalized Advantage Estimation

End-to-End Training of Deep Visuomotor Policies

Human-level control through deep reinforcement learning

Trust Region Policy Optimization

Adam: A Method for Stochastic Optimization

Evolving deep unsupervised convolutional networks for vision-based reinforcement learning

Playing Atari with Deep Reinforcement Learning

ADADELTA: An Adaptive Learning Rate Method

The Arcade Learning Environment: An Evaluation Platform for General Agents

Model-Free reinforcement learning with continuous action in practice

MapReduce for Parallel Reinforcement Learning

Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

Parallel reinforcement learning with linear function approximation

Neural Fitted Q Iteration - First Experiences with a Data Efficient Neural Reinforcement Learning Method

Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms

Introduction to Reinforcement Learning

Long Short-Term Memory

Learning from delayed rewards

Asynchronous Stochastic Approximation and Q-Learning

Incremental multi-step Q-learning

Asynchronous stochastic approximation and Q-learning

Simple statistical gradient-following algorithms for connectionist reinforcement learning

Distributed dynamic programming

MuJoCo: Modeling, Simulation and Visualization of Multi-Joint Dynamics with Contact (ed 1.0)

TORCS, The Open Racing Car Simulator, v1.3.5

TORCS, The Open Racing Car Simulator

Parallel and Distributed Evolutionary Algorithms: A Review

Reinforcement Learning: An Introduction

An Analysis of Temporal-Difference Learning with Function Approximation

, Geoffrey . Lecture 6 . 5 - rmsprop : Divide the gradient by a running average of its recent magnitude

On-line Q-learning using connectionist systems

On the Convergence of Stochastic Iterative Dynamic Programming Algorithms

Reinforcement Learning for Robots Using Neural Networks

Function Optimization using Connectionist Reinforcement Learning Algorithms

Field of Study

Computer ScienceMathematics

Journal Information

Name

ArXiv

Volume

abs/1602.01783

Venue Information

Name

International Conference on Machine Learning

Type

conference

URL

https://icml.cc/

Alternate Names

ICML
Int Conf Mach Learn