Benchmarks for Deep Off-Policy Evaluation

Published in

International Conference on Learning Representa...(2021)

External Links:

Generate Graph

TL;DR

The goal of the benchmark is to provide a standardized measure of progress that is motivated from a set of principles designed to challenge and test the limits of existing OPE methods.

Abstract

Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both obtaining and selecting complex policies for decision making. The ability to perform evaluation offline is particularly important in many real-world domains, such as healthcare, recommender systems, or robotics, where online data collection is an expensive and potentially dangerous process. Being able to accurately evaluate and select high-performing policies without requiring online interaction could yield significant benefits in safety, time, and cost for these applications. While many OPE methods have been proposed in recent years, comparing results between works is difficult because there is currently a lack of a comprehensive and unified benchmark. Moreover, it is difficult to measure how far algorithms have progressed, due to the lack of challenging evaluation tasks. In order to address this gap, we propose a new benchmark for off-policy evaluation which includes tasks on a range of challenging, high-dimensional control problems, with wide selections of datasets and policies for performing policy selection. The goal of of our benchmark is to provide a standardized measure of progress that is motivated from a set of principles designed to challenge and test the limits of existing OPE methods. We perform a comprehensive evaluation of state-of-the-art algorithms, and we will provide open-source access to all data and code to foster future research in this area.

Authors

Mohammad Norouzi

14 papers

T. Paine

5 papers

S. Levine

59 papers

References62 items

Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization

Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation

Hyperparameter Selection for Offline Reinforcement Learning

Off-Policy Evaluation via the Regularized Lagrangian

Critic Regularized Regression

Benchmarks for Deep Off-Policy Evaluation

Published in

International Conference on Learning Representa...(2021)

External Links:

Generate Graph

TL;DR

The goal of the benchmark is to provide a standardized measure of progress that is motivated from a set of principles designed to challenge and test the limits of existing OPE methods.

Abstract

Authors

Mohammad Norouzi

14 papers

T. Paine

5 papers

S. Levine

59 papers

References62 items

Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization

Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation

Hyperparameter Selection for Offline Reinforcement Learning

Off-Policy Evaluation via the Regularized Lagrangian

Critic Regularized Regression

Aviral Kumar

6 papers

Ofir Nachum

6 papers

Yutian Chen

2 papers

Alexander Novikov

5 papers

Cosmin Paduraru

3 papers

Mengjiao Yang

1 papers

Michael R. Zhang

1 papers

RL Unplugged: Benchmarks for Offline Reinforcement Learning

Acme: A Research Framework for Distributed Reinforcement Learning

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Batch Stationary Distribution Estimation

Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning

Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning

Grandmaster level in StarCraft II using multi-agent reinforcement learning

When to Trust Your Model: Model-Based Policy Optimization

Off-Policy Evaluation via Off-Policy Classification

DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections

Learning When-to-Treat Policies

Batch Policy Learning under Constraints

Off-Policy Evaluation and Learning from Logged Bandit Feedback: Error Reduction via Surrogate Policy

Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters

Learning to Drive in a Day

QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

Importance Sampling Policy Evaluation with an Estimated Behavior Policy

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Addressing Function Approximation Error in Actor-Critic Methods

Offline A/B Testing for Recommender Systems

Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations

Importance Sampling for Fair Policy Selection

Proximal Policy Optimization Algorithms

Optimal and Adaptive Off-policy Evaluation in Contextual Bandits

Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates

Safe and Efficient Off-Policy Reinforcement Learning

Off-policy evaluation for slate recommendation

Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning

Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

Continuous control with deep reinforcement learning

Personalized Ad Recommendation Systems for Life-Time Value Optimization with Guarantees

An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning

Trust Region Policy Optimization

High-Confidence Off-Policy Evaluation

Doubly Robust Policy Evaluation and Optimization

Offline policy evaluation across representations with applications to educational games

Playing Atari with Deep Reinforcement Learning

Reinforcement learning in robotics: A survey

MuJoCo: A physics engine for model-based control

Counterfactual reasoning and learning systems: the example of computational advertising

The Arcade Learning Environment: An Evaluation Platform for General Agents

The conference paper

A contextual-bandit approach to personalized news article recommendation

Fast gradient-descent methods for temporal-difference learning with linear function approximation

Evaluation of Policy Gradient Methods and Variants on the Cart-Pole Benchmark

Agent Based Decision Support System Using Reinforcement Learning Under Emergency Circumstances

Scaling Reinforcement Learning toward RoboCup Soccer

Eligibility Traces for Off-Policy Policy Evaluation

Planning treatment of ischemic heart disease with partially observable Markov decision processes

Temporal Difference Learning and TD-Gammon

Distributional policy gradients

Recommender systems , volume 1

Counterfactual risk minimization: Learning from logged bandit feedback

Fitted Policy Search: Direct Policy Search using a Batch Reinforcement Learning Approach

Planning with approximate and learned models of markov decision processes

Reinforcement Learning Benchmarks and Bake-offs II A workshop at the 2005 NIPS conference

Field of Study

Computer ScienceMathematics

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

International Conference on Learning Representations

Type

conference

URL

https://iclr.cc/

Alternate Names

Int Conf Learn Represent
ICLR