Value Prediction Network (2017-07-11T00:00:00.000000Z)

TL;DR

This paper proposes a novel deep reinforcement learning architecture, called Value Prediction Network (VPN), which integrates model-free and model-based RL methods into a single neural network, which outperforms Deep Q-Network on several Atari games even with short-lookahead planning.

Abstract

This paper proposes a novel deep reinforcement learning (RL) architecture, called Value Prediction Network (VPN), which integrates model-free and model-based RL methods into a single neural network. In contrast to typical model-based RL methods, VPN learns a dynamics model whose abstract states are trained to make option-conditional predictions of future values (discounted sum of rewards) rather than of future observations. Our experimental results show that VPN has several advantages over both model-free and model-based baselines in a stochastic environment where careful planning is required but building an accurate observation-prediction model is difficult. Furthermore, VPN outperforms Deep Q-Network (DQN) on several Atari games even with short-lookahead planning, demonstrating its potential as a new way of learning a good state representation.

Authors

Honglak Lee

21 papers

Junhyuk Oh

1 papers

Satinder Singh

1 papers

TL;DR

Abstract

Authors

References40 items

Recurrent Environment Simulators

Prediction and Control with Temporal Segment Models

Neural Map: Structured Memory for Deep Reinforcement Learning

Dynamic Action Repetition for Deep Reinforcement Learning

The Predictron: End-To-End Learning and Planning

Reinforcement Learning with Unsupervised Auxiliary Tasks

Deep visual foresight for planning robot motion

Video Pixel Networks

Strategic Attentive Writer for Learning Macro-Actions

Deep Successor Reinforcement Learning

Control of Memory, Active Perception, and Action in Minecraft

Unsupervised Learning for Physical Interaction through Video Prediction

Deep Learning for Reward Design to Improve Monte Carlo Tree Search in ATARI Games

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Continuous Deep Q-Learning with Model-based Acceleration

Value Iteration Networks

Asynchronous Methods for Deep Reinforcement Learning

Mastering the game of Go with deep neural networks and tree search

Deep Residual Learning for Image Recognition

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Dueling Network Architectures for Deep Reinforcement Learning

Learning Continuous Control Policies by Stochastic Value Gradients

Action-Conditional Video Prediction using Deep Networks in Atari Games

Deep Recurrent Q-Learning for Partially Observable MDPs

DeepMPC: Learning Deep Latent Features for Model Predictive Control

Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Human-level control through deep reinforcement learning

Adam: A Method for Stochastic Optimization

The Arcade Learning Environment: An Evaluation Platform for General Agents

Temporal-difference search in computer Go

A Survey of Monte Carlo Tree Search Methods

Variational Bayesian learning of nonlinear hidden state-space models for model predictive control

Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping

Bandit Based Monte-Carlo Planning

Learning Options in Reinforcement Learning

Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

Q-learning

Multi-Step Dyna Planning for Policy Evaluation and Control

Temporal abstraction in reinforcement learning

Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming

Field of Study

Venue Information

Name

Type

URL

Alternate Names