Dungeons and Data: A Large-Scale NetHack Dataset (2022-11-01T00:00:00.000000Z)

TL;DR

A wide range of existing algorithms are evaluated including online and offline RL, as well as learning from demonstrations, showing that significant research advances are needed to fully leverage large-scale datasets for challenging sequential decision making tasks.

Abstract

Recent breakthroughs in the development of agents to solve challenging sequential decision making problems such as Go, StarCraft, or DOTA, have relied on both simulated environments and large-scale datasets. However, progress on this research has been hindered by the scarcity of open-sourced datasets and the prohibitive computational cost to work with them. Here we present the NetHack Learning Dataset (NLD), a large and highly-scalable dataset of trajectories from the popular game of NetHack, which is both extremely challenging for current methods and very fast to run. NLD consists of three parts: 10 billion state transitions from 1.5 million human trajectories collected on the NAO public NetHack server from 2009 to 2020; 3 billion state-action-score transitions from 100,000 trajectories collected from the symbolic bot winner of the NetHack Challenge 2021; and, accompanying code for users to record, load and stream any collection of such trajectories in a highly compressed form. We evaluate a wide range of existing algorithms including online and offline RL, as well as learning from demonstrations, showing that significant research advances are needed to fully leverage large-scale datasets for challenging sequential decision making tasks.

Authors

Naila Murray

2 papers

R. Raileanu

5 papers

Tim Rocktäschel

10 papers

TL;DR

Abstract

Authors

References90 items

Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning

You Can't Count on Luck: Why Decision Transformers Fail in Stochastic Environments

Insights From the NeurIPS 2021 NetHack Challenge

A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems

Adversarially Trained Actor Critic for Offline Reinforcement Learning

RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning

Offline Reinforcement Learning with Implicit Q-Learning

BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning

MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning

The MAGICAL Benchmark for Robust Imitation

Learning quadrupedal locomotion over challenging terrain

Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning

Conservative Q-Learning for Offline Reinforcement Learning

The NetHack Learning Environment

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning

Dota 2 with Large Scale Deep Reinforcement Learning

Grandmaster level in StarCraft II using multi-agent reinforcement learning

RoboNet: Large-Scale Multi-Robot Learning

Solving Rubik's Cube with a Robot Hand

A Framework for Data-Driven Robotics

Behavior Regularized Offline Reinforcement Learning

MineRL: A Large-Scale Dataset of Minecraft Demonstrations

An Optimistic Perspective on Offline Reinforcement Learning

Striving for Simplicity in Off-policy Deep Reinforcement Learning

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog

Simitate: A Hybrid Imitation Learning Benchmark

Learning ambidextrous robot grasping policies

Off-Policy Deep Reinforcement Learning without Exploration

Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation

Behavioral Cloning from Observation

Imitating Latent Policies from Observation

Kickstarting Deep Reinforcement Learning

Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

StarCraft II: A New Challenge for Reinforcement Learning

Proximal Policy Optimization Algorithms

Emergence of Locomotion Behaviours in Rich Environments

Third-Person Imitation Learning

Predictive Off-Policy Policy Evaluation for Nonstationary Decision Problems, with Applications to Digital Marketing

Sim-to-Real Robot Learning from Pixels with Progressive Nets

Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection

Mastering the game of Go with deep neural networks and tree search

Playing Atari with Deep Reinforcement Learning

Vision meets robotics: The KITTI dataset

Sample-efficient batch reinforcement learning for dialogue management optimization

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Learning from Logged Implicit Exploration Data

A contextual-bandit approach to personalized news article recommendation

Hybrid Reinforcement/Supervised Learning of Dialogue Policies from Fixed Data Sets

Imitation learning from pixel observations for continuous control

The Telegraph

RL Unplugged: A Collection of Benchmarks for Offline Reinforcement Learning

NetHack Home Page

NetHack Wiki

Kenneth Lorber

The atari grand challenge

conduct (str) -a bitfield encoding the conducts 19 completed in the game

(str) -The name of the dataset to load

The number of steps to unroll for each episode in a minibatch

dbfilename -(str) -The path to the database file used

uid (int) -An id used (in conjunction with player name) to identify players (for save games etc)

Object) -An executor class that has a map function

The number of simulataneous episodes to load (i.e. size B)

gameids - (List[str]) - A predetermined list of gameids to iterate through

loop_forever - (bool) - Whether to start again at the beginning of gameids when the iterator runs out.

c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We report standard deviations across seeds for all results

b) URL to website/platform where the dataset/benchmark can be viewed and downloaded by the reviewers

e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We discuss

c) Author statement that they bear all responsibility in case of violation of rights, etc., and conﬁrmation of the data license. [We provide this in

turns (int)

ensure accessibility

Explicit license: Authors must choose a license, ideally a CC license for datasets, or an open source license for code (e.g. RL environments). [We

moolib: A Platform for Distributed RL