Deep Reinforcement Learning from Human Preferences (2017-06-12T00:00:00.000000Z)

TL;DR

This work explores goals defined in terms of (non-expert) human preferences between pairs of trajectory segments in order to effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion.

Abstract

For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.

Authors

P. Christiano

2 papers

Jan Leike

6 papers

Tom B. Brown

6 papers

TL;DR

Abstract

Authors

References45 items

Deep Q-learning From Demonstrations

Learning from Demonstrations for Real World Reinforcement Learning

Third-Person Imitation Learning

Interactive Learning from Policy-Dependent Human Feedback

Generalizing Skills with Semi-Supervised Reinforcement Learning

Breeding a diversity of Super Mario behaviors through interactive evolution

Concrete Problems in AI Safety

Generative Adversarial Imitation Learning

Learning Language Games through Interaction

Cooperative Inverse Reinforcement Learning

Should We Fear Supersmart Robots?

Score-based Inverse Reinforcement Learning

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

Model-Free Preference-Based Reinforcement Learning

Asynchronous Methods for Deep Reinforcement Learning

Mastering the game of Go with deep neural networks and tree search

Active reward learning with a novel acquisition function

Human-level control through deep reinforcement learning

Trust Region Policy Optimization

Active Reward Learning

Superintelligence: Paths, Dangers, Strategies

Programming by Feedback

Playing Atari with Deep Reinforcement Learning

Learning non-myopically from human-generated reward

MuJoCo: A physics engine for model-based control

A Bayesian Approach for Policy Learning from Trajectory Preference Queries

Learning from human-generated reward

Preference-based reinforcement learning: a formal framework and a policy iteration algorithm

APRIL: Active Preference-learning based Reinforcement Learning

The Arcade Learning Environment: An Evaluation Platform for General Agents

Preference-Based Policy Learning

Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning

A Bayesian interactive optimization approach to procedural animation design

Interactively shaping agents via human reinforcement: the TAMER framework

Picbreeder: evolving pictures collaboratively online

Algorithms for Inverse Reinforcement Learning

Individual Choice Behavior: A Theoretical Analysis

Stimulus and response generalization: A stochastic model relating generalization to distance in psychological space

RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS THE METHOD OF PAIRED COMPARISONS

Preference-Based Reinforcement Learning: A Preliminary Survey

Preference-learning based Inverse Reinforcement Learning for Dialog Control

INTRODUCING MACHINE LEARNING WITHIN AN INTERACTIVE EVOLUTIONARY DESIGN ENVIRONMENT

The rating of chessplayers, past and present

RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names