Turning the Tables: Biased, Imbalanced, Dynamic Tabular Datasets for ML Evaluation (2022-11-24T00:00:00.000000Z)

TL;DR

Bank Account Fraud is presented, the first publicly available privacy-preserving, large-scale, realistic suite of tabular datasets, and aims to provide the research community with a more realistic, complete, and robust test bed to evaluate novel and existing methods.

Abstract

Evaluating new techniques on realistic datasets plays a crucial role in the development of ML research and its broader adoption by practitioners. In recent years, there has been a significant increase of publicly available unstructured data resources for computer vision and NLP tasks. However, tabular data -- which is prevalent in many high-stakes domains -- has been lagging behind. To bridge this gap, we present Bank Account Fraud (BAF), the first publicly available privacy-preserving, large-scale, realistic suite of tabular datasets. The suite was generated by applying state-of-the-art tabular data generation techniques on an anonymized,real-world bank account opening fraud detection dataset. This setting carries a set of challenges that are commonplace in real-world applications, including temporal dynamics and significant class imbalance. Additionally, to allow practitioners to stress test both performance and fairness of ML methods, each dataset variant of BAF contains specific types of data bias. With this resource, we aim to provide the research community with a more realistic, complete, and robust test bed to evaluate novel and existing methods.

Authors

Sérgio Jesus

2 papers

Pedro Saleiro

3 papers

P. Bizarro

4 papers

TL;DR

Abstract

Authors

References55 items

Understanding Unfairness in Fraud Detection through Model and Data Bias Interactions

Prisoners of Their Own Devices: How Models Induce Data Bias in Performative Prediction

Algorithmic fairness datasets: the story so far

Benchmarking Differentially Private Synthetic Data Generation Algorithms

Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research

A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication

A survey on datasets for fairness‐aware machine learning

Retiring Adult: New Datasets for Fair Machine Learning

Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks

How Data Can Be Used Against People: A Classification of Personal Data Misuses

It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks

An Empirical Comparison of Bias Reduction Methods on Real-World Problems in High-Stakes Policy Settings

Pros and Cons of GAN Evaluation Measures: New Developments

The AI Index 2021 Annual Report

CTAB-GAN: Effective Table Data Synthesizing

Fairness in Machine Learning

Data and its (dis)contents: A survey of dataset development and use in machine learning research

Differentially Private Synthetic Data: Applied Evaluations and Enhancements

Performative Prediction

Value-laden disciplinary shifts in machine learning

Modeling Tabular data using Conditional GAN

ProPublica's COMPAS Data Revisited

Differentially Private Data Generative Models

PATE-GAN: Generating Synthetic Data with Differential Privacy Guarantees

Datasheets for datasets

Differentially Private Generative Adversarial Network

A comparative study of fairness-enhancing interventions in machine learning

LightGBM: A Highly Efficient Gradient Boosting Decision Tree

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

The Ugly Truth About Ourselves and Our Robot Creations: The Problem of Bias and Social Inequity

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Algorithmic Decision Making and the Cost of Fairness

Paysim: a financial mobile money simulator for fraud detection

False Positives, False Negatives, and False Analyses: A Rejoinder to "Machine Bias: There's Software Used across the Country to Predict Future Criminals. and It's Biased against Blacks"

Calibrating Probability with Undersampling for Unbalanced Classification

Fairness Constraints: Mechanisms for Fair Classification

Utilizing Noise Addition for Data Privacy, an Overview

The Unreasonable Effectiveness of Data

Differential Privacy: A Survey of Results

Calibrating Noise to Sensitivity in Private Data Analysis

Adversarial classification

Scaling Up the Accuracy of Naive-Bayes Classifiers: A Decision-Tree Hybrid

Patterns, predictions, and actions: Foundations of machine learning, chapter 8

Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual

OpenDP : An Open-Source Suite of Differential Privacy Tools

South german credit data: Correcting a widely used data set

and James Honaker

GENERATIVE ADVERSARIAL NETS

UCI machine learning repository

European Parliament and Council

c) Did you report error bars ( e.g. , with respect to the random seed after running experiments multiple times)?

c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Did you include the total amount of compute and the type of resources used ( e.g. , type of GPUs, internal cluster, or cloud provider)?

on the comparability of fees related to payment accounts, payment account switching and access to payment accounts with basic features

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names