BeBold: Exploration Beyond the Boundary of Explored Regions (2020-12-15T00:00:00.000000Z)

TL;DR

The regulated difference of inverse visitation counts is proposed as a simple but effective criterion for IR that helps the agent explore Beyond the Boundary of explored regions and mitigates common issues in count-based methods, such as short-sightedness and detachment.

Abstract

Efficient exploration under sparse rewards remains a key challenge in deep reinforcement learning. To guide exploration, previous work makes extensive use of intrinsic reward (IR). There are many heuristics for IR, including visitation counts, curiosity, and state-difference. In this paper, we analyze the pros and cons of each method and propose the regulated difference of inverse visitation counts as a simple but effective criterion for IR. The criterion helps the agent explore Beyond the Boundary of explored regions and mitigates common issues in count-based methods, such as short-sightedness and detachment. The resulting method, BeBold, solves the 12 most challenging procedurally-generated tasks in MiniGrid with just 120M environment steps, without any curriculum learning. In comparison, the previous SoTA only solves 50% of the tasks. BeBold also achieves SoTA on multiple tasks in NetHack, a popular rogue-like game that contains more challenging procedurally-generated environments.

Authors

K. Keutzer

18 papers

Joseph E. Gonzalez

10 papers

Xiaolong Wang

9 papers

TL;DR

Abstract

Authors

References58 items

Learning with AMIGo: Adversarially Motivated Intrinsic Goals

The NetHack Learning Environment

First return, then explore

Agent57: Outperforming the Atari Human Benchmark

RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments

Never Give Up: Learning Directed Exploration Strategies

Unsupervised Curricula for Visual Meta-Reinforcement Learning

Grandmaster level in StarCraft II using multi-agent reinforcement learning

Directed Exploration for Reinforcement Learning

Efficient Exploration via State Marginal Matching

Self-Supervised Exploration via Disagreement

Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration

Go-Explore: a New Approach for Hard-Exploration Problems

InfoBot: Transfer and Exploration via the Information Bottleneck

A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play

EMI: Exploration with Mutual Information

Model-Based Active Exploration

Exploration by Random Network Distillation

Large-Scale Study of Curiosity-Driven Learning

Self-Imitation Learning

Frontier Based Exploration for Autonomous Robot

Diversity is All You Need: Learning Skills without a Reward Function

Mastering the game of Go without human knowledge

Rainbow: Combining Improvements in Deep Reinforcement Learning

Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning

Reverse Curriculum Generation for Reinforcement Learning

Hindsight Experience Replay

GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms

Teacher–Student Curriculum Learning

Noisy Networks for Exploration

Parameter Space Noise for Exploration

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

Curiosity-Driven Exploration by Self-Supervised Prediction

Deep Exploration via Randomized Value Functions

Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

Count-Based Exploration with Neural Density Models

DeepStack: Expert-level artificial intelligence in heads-up no-limit poker

Variational Intrinsic Control

Reinforcement Learning with Unsupervised Auxiliary Tasks

Unifying Count-Based Exploration and Intrinsic Motivation

OpenAI Gym

VIME: Variational Information Maximizing Exploration

Deep Exploration via Bootstrapped DQN

Mastering the game of Go with deep neural networks and tree search

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

End-to-End Training of Deep Visuomotor Policies

Human-level control through deep reinforcement learning

Empowerment - an Introduction

PowerPlay: Training an Increasingly General Problem Solver by Continually Searching for the Simplest Still Unsolvable Problem

Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990–2010)

Curriculum learning

All Else Being Equal Be Empowered

Frontier-based exploration using multiple robots

A frontier-based approach for autonomous exploration

Preprint. Under review

Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies

OpenAI Pieter Abbeel, and Wojciech Zaremba

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names