CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning (2018-10-15T00:00:00.000000Z)

TL;DR

CURIOUS is proposed, an algorithm that leverages a modular Universal Value Function Approximator with hindsight learning to achieve a diversity of goals of different kinds within a unique policy and an automated curriculum learning mechanism that biases the attention of the agent towards goals maximizing the absolute learning progress.

Abstract

In open-ended environments, autonomous learning agents must set their own goals and build their own curriculum through an intrinsically motivated exploration. They may consider a large diversity of goals, aiming to discover what is controllable in their environments, and what is not. Because some goals might prove easy and some impossible, agents must actively select which goal to practice at any moment, to maximize their overall mastery on the set of learnable goals. This paper proposes CURIOUS, an algorithm that leverages 1) a modular Universal Value Function Approximator with hindsight learning to achieve a diversity of goals of different kinds within a unique policy and 2) an automated curriculum learning mechanism that biases the attention of the agent towards goals maximizing the absolute learning progress. Agents focus sequentially on goals of increasing complexity, and focus back on goals that are being forgotten. Experiments conducted in a new modular-goal robotic environment show the resulting developmental self-organization of a learning curriculum, and demonstrate properties of robustness to distracting goals, forgetting and changes in body properties.

Authors

Pierre-Yves Oudeyer

5 papers

Cédric Colas

4 papers

Olivier Sigaud

1 papers

TL;DR

Abstract

Authors

References47 items

Energy-Based Hindsight Experience Prioritization

Multi-task Deep Reinforcement Learning with PopArt

Curiosity Driven Exploration of Learned Disentangled Goal Spaces

Many-Goals Reinforcement Learning

Hierarchical Reinforcement Learning with Hindsight

Data-Efficient Hierarchical Reinforcement Learning

Learning by Playing - Solving Sparse Reward Tasks from Scratch

Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

Addressing Function Approximation Error in Actor-Critic Methods

Unicorn: Continual Learning with a Universal, Off-policy Agent

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Overcoming Exploration in Reinforcement Learning with Demonstrations

Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

Distral: Robust multitask reinforcement learning

Hindsight Experience Replay

GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms

Automatic Goal Generation for Reinforcement Learning Agents

FeUdal Networks for Hierarchical Reinforcement Learning

Modular active curiosity-driven discovery of tool use

OpenAI Gym

How Evolution May Work Through Curiosity-Driven Developmental Process

Prioritized Experience Replay

Continuous control with deep reinforcement learning

Universal Value Function Approximators

Playing Atari with Deep Reinforcement Learning

Active learning of inverse models with intrinsically motivated goal exploration in robots

The strategic student approach for life-long exploration and learning

Active choice of teachers, learning strategies and goals for a socially guided intrinsic motivation learner

The interaction of maturational constraints and intrinsic motivations in active motor development

Curious model-building control systems

On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other

Dataset

paper shows that a UVFA architecture trained with Deep Q-Network ( DQN ) can generalize to previously unseen goals

A bayesian foundation for individual learning under uncertainty

Maximizing Learning Progress: An Internal Reward System for Development

Reinforcement Learning: An Introduction

Learning to Achieve Goals

From the IMGEP perspective, a single monolithic modular multi-goal action policy

Empirical comparisons to other architectures: a goal-parameterized RL with HER (ﬂat multi-goal RL) and a multi-goal module-experts approach (one multi-goal expert per module)

the replay buffer

LP update

A study of the self-organization of learning phases demonstrated by our algorithm (automatic curriculum)

An environment for modular goal

a ′ where ϕ represents the parameters of the critic

efficient mechanism leveraging hindsight to efficiently in multi-goal settings

Experimental evidence of robustness to distracting goals , forgetting and body changes in comparison with random goal selection

a vector defined in space G .

Field of Study

Venue Information

Name

Type

URL

Alternate Names