Dueling Network Architectures for Deep Reinforcement Learning (2015-11-20T00:00:00.000000Z)

TL;DR

This paper presents a new neural network architecture for model-free reinforcement learning that leads to better policy evaluation in the presence of many similar-valued actions and enables the RL agent to outperform the state-of-the-art on the Atari 2600 domain.

Abstract

In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto-encoders. In this paper, we present a new neural network architecture for model-free reinforcement learning. Our dueling network represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. Our results show that this architecture leads to better policy evaluation in the presence of many similar-valued actions. Moreover, the dueling architecture enables our RL agent to outperform the state-of-the-art on the Atari 2600 domain.

Authors

Nando de Freitas

12 papers

H. V. Hasselt

5 papers

Ziyun Wang

6 papers

Dueling Network Architectures for Deep Reinforcement Learning

TL;DR

Abstract

Authors

References28 items

Mastering the game of Go with deep neural networks and tree search

Increasing the Action Gap: New Operators for Reinforcement Learning

Prioritized Experience Replay

Deep Reinforcement Learning with Double Q-Learning

Massively Parallel Methods for Deep Reinforcement Learning

Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

High-Dimensional Continuous Control Using Generalized Advantage Estimation

End-to-End Training of Deep Visuomotor Policies

Human-level control through deep reinforcement learning

Multiple Object Recognition with Visual Attention

Move Evaluation in Go Using Deep Convolutional Neural Networks

Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

Advances in optimizing recurrent networks

The Arcade Learning Environment: An Evaluation Platform for General Agents

Double Q-learning

A theoretical and empirical analysis of Expected Sarsa

Policy Gradient Methods for Reinforcement Learning with Function Approximation

Multi-Agent Residual Advantage Learning with General Function Approximation.

Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

Cognitron: A self-organizing multilayered neural network

Under Review as a Conference Paper at Iclr 2017 Delving into Transferable Adversarial Ex- Amples and Black-box Attacks

Deep Learning

Multi-player residual advantage learning with general function

Advantage Updating Applied to a Differrential Game

Reinforcement Learning for Robots Using Neural Networks

Single Clip performs better than Single

Field of Study

Venue Information

Name

Type

URL

Alternate Names