CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021-10-19T00:00:00.000000Z)

TL;DR

This work presents CORA, a platform for Continual Reinforcement Learning Agents that provides benchmarks, baselines, and metrics in a single code package, and hopes that the continual RL community can benefit from contributions, to accelerate the development of new continual RL algorithms.

Abstract

Progress in continual reinforcement learning has been limited due to several barriers to entry: missing code, high compute requirements, and a lack of suitable benchmarks. In this work, we present CORA, a platform for Continual Reinforcement Learning Agents that provides benchmarks, baselines, and metrics in a single code package. The benchmarks we provide are designed to evaluate different aspects of the continual RL challenge, such as catastrophic forgetting, plasticity, ability to generalize, and sample-efficient learning. Three of the benchmarks utilize video game environments (Atari, Procgen, NetHack). The fourth benchmark, CHORES, consists of four different task sequences in a visually realistic home simulator, drawn from a diverse set of task and scene parameters. To compare continual RL methods on these benchmarks, we prepare three metrics in CORA: Continual Evaluation, Isolated Forgetting, and Zero-Shot Forward Transfer. Finally, CORA includes a set of performant, open-source baselines of existing algorithms for researchers to use and expand on. We release CORA and hope that the continual RL community can benefit from our contributions, to accelerate the development of new continual RL algorithms.

Authors

A. Gupta

21 papers

Roozbeh Mottaghi

15 papers

Eric Kolve

7 papers

TL;DR

Abstract

Authors

References90 items

EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine

Avalanche RL: a Continual Reinforcement Learning Library

MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research

Wanderlust: Online Continual Object Detection in the Real World

Sequoia: A Software Framework to Unify Continual Learning Research

Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Continual World: A Robotic Benchmark For Continual Reinforcement Learning

ManipulaTHOR: A Framework for Visual Object Manipulation

Avalanche: an End-to-End Library for Continual Learning

Continuous Coordination As a Realistic Scenario for Lifelong Learning

The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors

Online Continual Learning in Image Classification: An Empirical Survey

Towards Continual Reinforcement Learning: A Review and Perspectives

iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes

Rearrangement: A Challenge for Embodied AI

Prioritized Level Replay

robosuite: A Modular Simulation Framework and Benchmark for Robot Learning

A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

Transient Non-stationarity and Generalisation in Deep Reinforcement Learning

The NetHack Learning Environment

SAPIEN: A SimulAted Part-Based Interactive ENvironment

Jelly Bean World: A Testbed for Never-Ending Learning

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

Leveraging Procedural Generation to Benchmark Reinforcement Learning

IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks

Grandmaster level in StarCraft II using multi-agent reinforcement learning

Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

TorchBeast: A PyTorch Platform for Distributed RL

RLBench: The Robot Learning Benchmark & Learning Environment

Behaviour Suite for Reinforcement Learning

Meta-Learning Representations for Continual Learning

Continual Reinforcement Learning in 3D Non-stationary Environments

Habitat: A Platform for Embodied AI Research

VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning

Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning

Experience Replay for Continual Learning

Environments for Lifelong Reinforcement Learning

Don't forget, there is more than forgetting: new metrics for Continual Learning

Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines

Generalization and Regularization in DQN

VirtualHome: Simulating Household Activities Via Programs

Progress & Compress: A scalable framework for continual learning

A Study on Overfitting in Deep Reinforcement Learning

Selective Experience Replay for Lifelong Learning

Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

Continual Lifelong Learning with Neural Networks: A Review

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence

CHALET: Cornell House Agent Learning Environment

DeepMind Control Suite

AI2-THOR: An Interactive 3D Environment for Visual AI

MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

HoME: a Household Multimodal Environment

Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

StarCraft II: A New Challenge for Reinforcement Learning

Gradient Episodic Memory for Continual Learning

DeepMind Lab

Overcoming catastrophic forgetting in neural networks

Target-driven visual navigation in indoor scenes using deep reinforcement learning

OpenAI Gym

ViZDoom: A Doom-based AI research platform for visual reinforcement learning

Reinforcement learning vs human programming in tetherball robot games

Human-level control through deep reinforcement learning

Contextual Markov Decision Processes

Never-Ending Learning

NEIL: Extracting Visual Knowledge from Web Data

Reinforcement learning in robotics: A survey

A Survey on Policy Search for Robotics

The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects

Reinforcement Learning in Robotics: Applications and Real-World Challenges

ELLA: An Efficient Lifelong Learning Algorithm