Large-Scale Study of Curiosity-Driven Learning (2018-08-13T00:00:00.000000Z)

TL;DR

This paper performs the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite, and shows surprisingly good performance.

Abstract

Reinforcement learning algorithms rely on carefully engineering environment rewards that are extrinsic to the agent. However, annotating each environment with hand-designed, dense rewards is not scalable, motivating the need for developing reward functions that are intrinsic to the agent. Curiosity is a type of intrinsic reward function which uses prediction error as reward signal. In this paper: (a) We perform the first large-scale study of purely curiosity-driven learning, i.e. without any extrinsic rewards, across 54 standard benchmark environments, including the Atari game suite. Our results show surprisingly good performance, and a high degree of alignment between the intrinsic curiosity objective and the hand-designed extrinsic rewards of many game environments. (b) We investigate the effect of using different feature spaces for computing prediction error and show that random features are sufficient for many popular RL game benchmarks, but learned features appear to generalize better (e.g. to novel game levels in Super Mario Bros.). (c) We demonstrate limitations of the prediction-based rewards in stochastic setups. Game-play videos and code are at this https URL

Authors

Alexei A. Efros

23 papers

A. Storkey

6 papers

Trevor Darrell

47 papers

TL;DR

Abstract

Authors

References48 items

Computational Theories of Curiosity-Driven Learning

Diversity is All You Need: Learning Skills without a Reward Function

Zero-Shot Visual Imitation

UCB EXPLORATION VIA Q-ENSEMBLES

Proximal Policy Optimization Algorithms

Noisy Networks for Exploration

Parameter Space Noise for Exploration

Curiosity-Driven Exploration by Self-Supervised Prediction

Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play

Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

Count-Based Exploration with Neural Density Models

EX2: Exploration with Exemplar Models for Deep Reinforcement Learning

Variational Intrinsic Control

#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

Unifying Count-Based Exploration and Intrinsic Motivation

VIME: Variational Information Maximizing Exploration

Deep Exploration via Bootstrapped DQN

Why Greatness Cannot Be Planned: The Myth of the Objective

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Human-level control through deep reinforcement learning

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Deep Fried Convnets

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress

The Arcade Learning Environment: An Evaluation Platform for General Agents

An information-theoretic approach to curiosity-driven reinforcement learning

The role of Game Discourse Analysis and curiosity in creating engaging and effective serious games by implementing a back story and foreshadowing

On Random Weights and Unsupervised Feature Learning

Abandoning Objectives: Evolution Through the Search for Novelty Alone

Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010)

What is the best multi-stage architecture for object recognition?

Et al

What is Intrinsic Motivation? A Typology of Computational Approaches

An analytic solution to discrete Bayesian reinforcement learning

Intrinsically Motivated Reinforcement Learning

Curious model-building control systems

A possibility for implementing curiosity and boredom in model-building neural controllers

Uncertainty in Games

and E

Exploiting Open-Endedness to Solve Problems Through the Search for Novelty

The Development of Embodied Cognition: Six Lessons from Babies

MDA : A Formal Approach to Game Design and Game Research

Why we play games: Four keys to more emotion in player experiences

Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions.

Richard; Deci

Reinforcement Learning: An Introduction

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names