KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems (2022-02-22T00:00:00.000000Z)

TL;DR

A fully-observed dataset from Kuaishou's online environment, where almost all 1,411 users have been exposed to all 3,327 items, and a preliminary analysis of how the two factors - data density and exposure bias - affect the evaluation results of multi-round conversational recommendation.

Abstract

The progress of recommender systems is hampered mainly by evaluation as it requires real-time interactions between humans and systems, which is too laborious and expensive. This issue is usually approached by utilizing the interaction history to conduct offline evaluation. However, existing datasets of user-item interactions are partially observed, leaving it unclear how and to what extent the missing interactions will influence the evaluation. To answer this question, we collect a fully-observed dataset from Kuaishou's online environment, where almost all 1,411 users have been exposed to all 3,327 items. To the best of our knowledge, this is the first real-world fully-observed data with millions of user-item interactions. With this unique dataset, we conduct a preliminary analysis of how the two factors - data density and exposure bias - affect the evaluation results of multi-round conversational recommendation. Our main discoveries are that the performance ranking of different methods varies with the two factors, and this effect can only be alleviated in certain cases by estimating missing interactions for user simulation. This demonstrates the necessity of the fully-observed dataset. We release the dataset and the pipeline implementation for evaluation at https://kuairec.com

Authors

Xiangnan He

12 papers

Tat-seng Chua

9 papers

Wenqiang Lei

4 papers

TL;DR

Abstract

Authors

References77 items

Variational Reasoning about User Preferences for Conversational Recommendation

User-Centric Conversational Recommendation with Multi-Aspect User Modeling

CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender System

Who are the Best Adopters? User Selection Model for Free Trial Item Promotion

C²-CRS: Coarse-to-Fine Contrastive Learning for Conversational Recommender System

Multiple Choice Questions based Multi-Interest Policy Learning for Conversational Recommendation

DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation

RecSys 2021 Tutorial on Conversational Recommendation: Formulation, Methods, and Evaluation

Counterfactual Data-Augmented Sequential Recommendation

Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning

Causal Intervention for Leveraging Popularity Bias in Recommendation

Adapting User Preference to Online Feedback in Multi-round Conversational Recommendation

Information Management in Smart Cities: Turning end users' views into multi-item scale development, validation, and policy-making recommendations

Advances and Challenges in Conversational Recommender Systems: A Survey

Towards Topic-Guided Conversational Recommender System

Bias and Debias in Recommender System: A Survey and Future Directions

Keeping Dataset Biases out of the Simulation: A Debiased Simulator for Reinforcement Learning based Recommender Systems

"Click" Is Not Equal to "Like": Counterfactual Recommendation for Mitigating Clickbait Issue

S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization

Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation

A General Knowledge Distillation Framework for Counterfactual Recommendation via Uniform Data

Deep Critiquing for VAE-based Recommender Systems

Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusion

Interactive Path Reasoning on Graph for Conversational Recommendation

Multi-sided Exposure Bias in Recommendation

Evaluating Conversational Recommender Systems via User Simulation

Self-Supervised Reinforcement Learning for Recommender Systems

Seamlessly Unifying Attributes and Items: Conversational Recommendation for Cold-start Users

Towards Conversational Recommendation over Multi-Type Dialogs

Reward Constrained Interactive Recommendation with Natural Language Feedback

Latent Linear Critiquing for Conversational Recommender Systems

A Survey on Conversational Recommender Systems

Pseudo Dyna-Q: A Reinforcement Learning Framework for Interactive Recommendation

Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems

Learning Disentangled Representations for Recommendation

RecSim: A Configurable Simulation Platform for Recommender Systems

Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback

Generating Reliable Friends via Adversarial Training to Improve Social Recommendation

Towards Knowledge-Based Recommender Dialog System

Reinforcement Knowledge Graph Reasoning for Explainable Recommendation

Conversational Contextual Bandit: Algorithm and Application

Strategyproof multi-item exchange under single-minded dichotomous preferences

BLOMA: Explain Collaborative Filtering via Boosted Local Rank-One Matrix Approximation

When People Change their Mind: Off-Policy Evaluation in Non-stationary Recommendation Environments

Large-scale Interactive Recommendation with Tree-structured Policy Gradient

Towards Deep Conversational Recommendations

Adaptive Implicit Friends Identification over Heterogeneous Network for Social Recommendation

Towards Conversational Search and Recommendation: System Ask, User Respond

Q&R: A Two-Stage Approach toward Interactive Recommendation

Conversational Recommender System

DRN: A Deep Reinforcement Learning Framework for News Recommendation

Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning

Offline A/B Testing for Recommender Systems

Online Interactive Collaborative Filtering Using Multi-Armed Bandit with Dependent Arms

Deal or No Deal? End-to-End Learning of Negotiation Dialogues

Large-scale Validation of Counterfactual Learning Methods: A Test-Bed

Towards Conversational Recommender Systems

Recommendations as Treatments: Debiasing Learning and Evaluation

Modeling User Exposure in Recommendation

Toward Predicting the Outcome of an A/B Experiment for Search Relevance

Reducing Offline Evaluation Bias in Recommendation Systems

Interactive collaborative filtering

Ranking with non-random missing ratings: influence of popularity and positivity on evaluation metrics

Challenging the Long Tail Recommendation

Critiquing-based recommenders: survey and emerging trends

Factorization Machines

Collaborative prediction and ranking with non-random missing data

The long tail of recommender systems and how to leverage it

Probabilistic Matrix Factorization

Collaborative Filtering and the Missing at Random Assumption

A Modern Introduction to Probability and Statistics: Understanding Why and How

Power laws, Pareto distributions and Zipf's law

Evaluating Kolmogorov's distribution

Statistical Analysis With Missing Data

Counterfactual Risk Minimization: Learning from Logged Bandit Feedback

IR evaluation methods for retrieving highly relevant documents