Revisiting Fundamentals of Experience Replay (2020-07-12T00:00:00.000000Z)

TL;DR

This work presents a systematic and extensive analysis of experience replay in Q-learning methods, focusing on two fundamental properties: the replay capacity and the ratio of learning updates to experience collected (replay ratio).

Abstract

Experience replay is central to off-policy algorithms in deep reinforcement learning (RL), but there remain significant gaps in our understanding. We therefore present a systematic and extensive analysis of experience replay in Q-learning methods, focusing on two fundamental properties: the replay capacity and the ratio of learning updates to experience collected (replay ratio). Our additive and ablative studies upend conventional wisdom around experience replay -- greater capacity is found to substantially increase the performance of certain algorithms, while leaving others unaffected. Counterintuitively we show that theoretically ungrounded, uncorrected n-step returns are uniquely beneficial while other techniques confer limited benefit for sifting through larger memory. Separately, by directly controlling the replay ratio we contextualize previous observations in the literature and empirically measure its importance across a variety of deep RL algorithms. Finally, we conclude by testing a set of hypotheses on the nature of these performance benefits.

Authors

H. Larochelle

18 papers

Yoshua Bengio

69 papers

Will Dabney

5 papers

TL;DR

Abstract

Authors

References59 items

Attentive Experience Replay

Combining Q-Learning and Search with Amortized Value Estimates

An Optimistic Perspective on Offline Reinforcement Learning

Ranking Policy Gradient

Experience Replay Optimization

When to use parametric models in reinforcement learning?

Importance Resampling for Off-policy Prediction

Diagnosing Bottlenecks in Deep Q-learning Algorithms

Hyperbolic Discounting and Learning over Multiple Horizons

Deep Reinforcement Learning and the Deadly Triad

An Introduction to Deep Reinforcement Learning

Dynamic Programming

Recurrent Experience Replay in Distributed Reinforcement Learning

Dopamine: A Research Framework for Deep Reinforcement Learning

Remember and Forget for Experience Replay

Organizing Experience: a Deeper Look at Replay Mechanisms for Sample-Based Planning in Continuous State Domains

Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update

Distributed Prioritized Experience Replay

A Deeper Look at Experience Replay

Distributional Reinforcement Learning with Quantile Regression

The Effects of Memory Replay in Reinforcement Learning

Rainbow: Combining Improvements in Deep Reinforcement Learning

Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

A Distributional Perspective on Reinforcement Learning

Noisy Networks for Exploration

Reinforcement Learning with Unsupervised Auxiliary Tasks

Sample Efficient Actor-Critic with Experience Replay

Safe and Efficient Off-Policy Reinforcement Learning

Unifying Count-Based Exploration and Intrinsic Motivation

Q(λ) with Off-Policy Corrections

Dueling Network Architectures for Deep Reinforcement Learning

Prioritized Experience Replay

Deep Reinforcement Learning with Double Q-Learning

Deep Recurrent Q-Learning for Partially Observable MDPs

Human-level control through deep reinforcement learning

Adam: A Method for Stochastic Optimization

A new Q(lambda) with interim forward view and Monte Carlo equivalence

Playing Atari with Deep Reinforcement Learning

The Arcade Learning Environment: An Evaluation Platform for General Agents

Algorithms for Reinforcement Learning

Eligibility Traces for Off-Policy Policy Evaluation

Reinforcement Learning: A Survey

Learning from delayed rewards

Incremental multi-step Q-learning

Markov Decision Processes: Discrete Stochastic Dynamic Programming

Q-learning

Self-improving reactive agents based on reinforcement learning, planning and teaching

Revisiting Fundamentals of Experience Replay

International Conference on Machine Learning (

Reconciling λ-Returns with Experience Replay

A new Q ( � ) with interim forward view and Monte Carlo equivalence

Box2D: A 2D physics engine for games

Neuro-Dynamic Programming

Reinforcement Learning: An Introduction

Rainbow Median Improvement Std. Deviation Replay Capacity

Figure 17. Rainbow replay buffer effects at a per-game level

Discovering that n -step returns are uniquely critical for taking advantage of an increased replay capacity

a) Disentangling the effects of replay capacity and old-est policy , ﬁnding that increasing replay capacity and decreasing the age of the oldest policy improves performance

c) Benchmarking n -step returns in the massive replay capacity regime, and ﬁnding that it still provides gains despite the substantial off-policyness of the data

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names