Quantifying Generalization in Reinforcement Learning (2018-12-06T00:00:00.000000Z)

TL;DR

It is shown that deeper convolutional architectures improve generalization, as do methods traditionally found in supervised learning, including L2 regularization, dropout, data augmentation and batch normalization.

Abstract

In this paper, we investigate the problem of overfitting in deep reinforcement learning. Among the most common benchmarks in RL, it is customary to use the same environments for both training and testing. This practice offers relatively little insight into an agent's ability to generalize. We address this issue by using procedurally generated environments to construct distinct training and test sets. Most notably, we introduce a new environment called CoinRun, designed as a benchmark for generalization in RL. Using CoinRun, we find that agents overfit to surprisingly large training sets. We then show that deeper convolutional architectures improve generalization, as do methods traditionally found in supervised learning, including L2 regularization, dropout, data augmentation and batch normalization.

Authors

Christopher Hesse

5 papers

John Schulman

14 papers

K. Cobbe

5 papers

Quantifying Generalization in Reinforcement Learning

TL;DR

Abstract

Authors

References20 items

Assessing Generalization in Deep Reinforcement Learning

Generalization and Regularization in DQN

Towards Understanding Regularization in Batch Normalization

Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation

A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning

AutoAugment: Learning Augmentation Policies from Data

A Study on Overfitting in Deep Reinforcement Learning

Gotta Learn Fast: A New Benchmark for Generalization in RL

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

Improved Regularization of Convolutional Neural Networks with Cutout

Proximal Policy Optimization Algorithms

Domain randomization for transferring deep neural networks from simulation to the real world

Human-level control through deep reinforcement learning

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Playing Atari with Deep Reinforcement Learning

The Arcade Learning Environment: An Evaluation Platform for General Agents

Long Short-Term Memory

The Impact of Determinism on Learning Atari 2600 Games

On the shortest spanning subtree of a graph and the traveling salesman problem

Field of Study

Venue Information

Name

Type

URL

Alternate Names