Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation (2020-08-17T00:00:00.000000Z)

TL;DR

Open Bandit Dataset is presented, a public logged bandit dataset collected on a large-scale fashion e-commerce platform, ZOZOTOWN, that enables experimental comparisons of different OPE estimators for the first time and develops Python software called Open Bandit Pipeline to streamline and standardize the implementation of batch bandit algorithms and OPE.

Abstract

Off-policy evaluation (OPE) aims to estimate the performance of hypothetical policies using data generated by a different policy. Because of its huge potential impact in practice, there has been growing research interest in this field. There is, however, no real-world public dataset that enables the evaluation of OPE, making its experimental studies unrealistic and irreproducible. With the goal of enabling realistic and reproducible OPE research, we present Open Bandit Dataset, a public logged bandit dataset collected on a large-scale fashion e-commerce platform, ZOZOTOWN. Our dataset is unique in that it contains a set of multiple logged bandit datasets collected by running different policies on the same platform. This enables experimental comparisons of different OPE estimators for the first time. We also develop Python software called Open Bandit Pipeline to streamline and standardize the implementation of batch bandit algorithms and OPE. Our open data and software will contribute to fair and transparent OPE research and help the community identify fruitful research directions. We provide extensive benchmark experiments of existing OPE estimators using our dataset and software. The results open up essential challenges and new avenues for future OPE research.

Authors

Yuta Saito

4 papers

Shunsuke Aihara

2 papers

Megumi Matsutani

1 papers

TL;DR

Abstract

Authors

References63 items

Counterfactual Learning and Evaluation for Recommender Systems: Foundations, Implementations, and Recent Advances

Evaluating the Robustness of Off-Policy Evaluation

Finite-time Analysis of Globally Nonstationary Multi-Armed Bandits

Benchmarks for Deep Off-Policy Evaluation

A Practical Guide of Off-Policy Evaluation for Bandit Problems

Optimal Off-Policy Evaluation from Multiple Logging Policies

Doubly Robust Estimator for Ranking Metrics with Post-Click Conversions

Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions

Open Graph Benchmark: Datasets for Machine Learning on Graphs

Off-Policy Evaluation and Learning for External Validity under a Covariate Shift

Debiased Off-Policy Evaluation for Recommendation Systems

Off-policy Bandit and Reinforcement Learning

Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning

Efficiently Breaking the Curse of Horizon: Double Reinforcement Learning in Infinite-Horizon Processes

Doubly robust off-policy evaluation with shrinkage

Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling

Off-Policy Evaluation via Off-Policy Classification

Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning

On the Design of Estimators for Bandit Off-Policy Evaluation

CAB: Continuous Adaptive Blending for Policy Evaluation and Learning

Offline Evaluation to Make Decisions About PlaylistRecommendation Algorithms

Adapting multi-armed bandits policies to contextual bandits scenarios

Efficient Counterfactual Learning from Bandit Feedback

Efficient Counterfactual Learning from Bandit Feedback

RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising

Representation Balancing MDPs for Off-Policy Policy Evaluation

Offline Evaluation of Ranking Policies with Click Models

Policy Evaluation and Optimization with Continuous Treatments

A Contextual Bandit Bake-off

More Robust Doubly Robust Off-policy Evaluation

Offline A/B Testing for Recommender Systems

Effective Evaluation Using Logged Bandit Feedback from Multiple Loggers

Optimal and Adaptive Off-policy Evaluation in Contextual Bandits

Large-scale Validation of Counterfactual Learning Methods: A Test-Bed

Off-policy evaluation for slate recommendation

Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning

Deep Residual Learning for Image Recognition

The Self-Normalized Estimator for Counterfactual Learning

Doubly Robust Off-policy Value Evaluation for Reinforcement Learning

High Confidence Policy Improvement

High-Confidence Off-Policy Evaluation

Doubly Robust Policy Evaluation and Optimization

Offline policy evaluation across representations with applications to educational games

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Counterfactual reasoning and learning systems: the example of computational advertising

Scikit-learn: Machine Learning in Python

Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms

Learning from Logged Implicit Exploration Data

A contextual-bandit approach to personalized news article recommendation

ImageNet: A large-scale hierarchical image database

The offset tree for learning with partial labels

Marginal Mean Models for Dynamic Regimes

Eligibility Traces for Off-Policy Policy Evaluation

ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES

Benchmarking Graph Neural Networks

Causal Combinatorial Factorization Machines for Set-Wise Recommendation

Ofﬂine Evaluation of Multi-Armed Bandit Algorithms Using Bootstrapped Replay on Expanded Data

provides scripts and instructions to reproduce the benchmark results using Open Bandit Dataset. We utilize tools such as Hydra 13 and Poetry 14 to ensure the reproducibility of the 12

* Linear/Logistic Epsilon Greedy * Linear/Logistic Thompson Sampling * Linear/Logistic Upper Confidence Bound • Offline (Batch) Bandit Algorithms

Batch learning from logged bandit feedback through counterfactual risk minimization

the estimated performance of BernoulliTS with the performance of Random # our OPE procedure suggests that BernoulliTS improves Random by 19.81% >>> ground_truth_random = bandit_feedback

Example Code (Bandit Experiment with Open Bandit Dataset)

Table 9 shows the benchmark results (RMSE) for the three campaigns when the OPE situation is Random → Bernoulli TS

Field of Study