Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning (2022-07-23T00:00:00.000000Z)

TL;DR

The experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems, and it is argued that utilising predefined skills provides a useful inductive bias for RL problems, especially those with large state-action spaces and sparse rewards.

Abstract

Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them. Instead, they are usually trained end-to-end, with the hope being that useful skills will be implicitly learned in order to maximise discounted return of some extrinsic reward function. In this paper, we investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments with large state-action spaces and sparse rewards. To this end, we created SkillHack, a benchmark of tasks and associated skills based on the game of NetHack. We evaluate a number of baselines on this benchmark, as well as our own novel skill-based method Hierarchical Kickstarting (HKS), which is shown to outperform all other evaluated methods. Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems. We ultimately argue that utilising predefined skills provides a useful inductive bias for RL problems, especially those with large state-action spaces and sparse rewards.

Authors

Mikayel Samvelyan

7 papers

Edward Grefenstette

14 papers

Tim Rocktaschel

5 papers

TL;DR

Abstract

Authors

References33 items

CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents

MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research

Benchmarking the Spectrum of Agent Capabilities

The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors

The NetHack Learning Environment

Effective Diversity in Population-Based Reinforcement Learning

Leveraging Procedural Generation to Benchmark Reinforcement Learning

TorchBeast: A PyTorch Platform for Distributed RL

Distilling Policy Distillation

Exploration by Random Network Distillation

Data-Efficient Hierarchical Reinforcement Learning

Kickstarting Deep Reinforcement Learning

Diversity is All You Need: Learning Skills without a Reward Function

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Learning Multi-Level Hierarchies with Hindsight

Population Based Training of Neural Networks

Meta Learning Shared Hierarchies

Distral: Robust multitask reinforcement learning

FeUdal Networks for Hierarchical Reinforcement Learning

The Option-Critic Architecture

ViZDoom: A Doom-based AI research platform for visual reinforcement learning

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning

Policy Distillation

Distilling the Knowledge in a Neural Network

The Arcade Learning Environment: An Evaluation Platform for General Agents

Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

Benchmarking

Feudal Reinforcement Learning

minerl

Exploiting Open-Endedness to Solve Problems Through the Search for Novelty

Actor-Critic Algorithms

The hyperparameters for the IMPALA agent are summarised in

Field of Study

Journal Information

Name

Volume