Addressing Function Approximation Error in Actor-Critic Methods (2018-02-26T00:00:00.000000Z)

TL;DR

This paper builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation, and draws the connection between target networks and overestimation bias.

Abstract

In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.

Authors

Scott Fujimoto

4 papers

D. Meger

6 papers

H. V. Hoof

2 papers

TL;DR

Abstract

Authors

References50 items

Dynamic Programming

Smoothed Action Value Functions for Learning Gaussian Policies

Distributed Prioritized Experience Replay

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Action Branching Architectures for Deep Reinforcement Learning

Deep Reinforcement Learning that Matters

The Uncertainty Bellman Equation and Exploration

Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation

Proximal Policy Optimization Algorithms

A Distributional Perspective on Reinforcement Learning

Discrete Sequential Prediction of Continuous Actions for Deep RL

Data-efficient Deep Reinforcement Learning for Dexterous Manipulation

Multi-step Off-policy Learning Without Importance Sampling Ratios

Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning

Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening

Safe and Efficient Off-Policy Reinforcement Learning

Continuous Deep Q-Learning with Model-based Acceleration

Deep Exploration via Bootstrapped DQN

Asynchronous Methods for Deep Reinforcement Learning

Taming the Noise in Reinforcement Learning via Soft Updates

Dueling Network Architectures for Deep Reinforcement Learning

Prioritized Experience Replay

Deep Reinforcement Learning with Double Q-Learning

Continuous control with deep reinforcement learning

Human-level control through deep reinforcement learning

Trust Region Policy Optimization

Adam: A Method for Stochastic Optimization

Deterministic Policy Gradient Algorithms

Bias-corrected Q-learning to control max-operator bias in Q-learning

MuJoCo: A physics engine for model-based control

Mean-Variance Optimization in Markov Decision Processes

Double Q-learning

A theoretical and empirical analysis of Expected Sarsa

Biasing Approximate Dynamic Programming with a Lower Discount Factor

Bias and Variance Approximation in Value Function Estimates

Off-Policy Temporal Difference Learning with Function Approximation

Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms

Dynamic Programming and Optimal Control. Volume 1

Reinforcement Learning: A Survey

Learning from delayed rewards

Dynamic Programming and Optimal Control, Two Volume Set

Self-improving reactive agents based on reinforcement learning, planning and teaching

Learning to predict by the methods of temporal differences

Distributional policy gradients

Issues in Using Function Approximation for Reinforcement Learning

Actor-Critic Algorithms

Reinforcement Learning: An Introduction

Estimator Variance in Reinforcement Learning: Theoretical Problems and Practical Solutions

On the theory of brownian motion

Field of Study

Venue Information

Name

Type

URL

Alternate Names