AQuA: A Benchmarking Tool for Label Quality Assessment (2023-06-15T00:00:00.000000Z)

TL;DR

A benchmarking environment AQuA is proposed to rigorously evaluate methods that enable machine learning in the presence of label noise and a design space is introduced to delineate concrete design choices of label error detection models.

Abstract

Machine learning (ML) models are only as good as the data they are trained on. But recent studies have found datasets widely used to train and evaluate ML models, e.g. ImageNet, to have pervasive labeling errors. Erroneous labels on the train set hurt ML models' ability to generalize, and they impact evaluation and model selection using the test set. Consequently, learning in the presence of labeling errors is an active area of research, yet this field lacks a comprehensive benchmark to evaluate these methods. Most of these methods are evaluated on a few computer vision datasets with significant variance in the experimental protocols. With such a large pool of methods and inconsistent evaluation, it is also unclear how ML practitioners can choose the right models to assess label quality in their data. To this end, we propose a benchmarking environment AQuA to rigorously evaluate methods that enable machine learning in the presence of label noise. We also introduce a design space to delineate concrete design choices of label error detection models. We hope that our proposed design space and benchmark enable practitioners to choose the right tools to improve their label quality and that our benchmark enables objective and rigorous evaluation of machine learning tools facing mislabeled data.

Authors

Arjun Choudhry

2 papers

Mononito Goswami

1 papers

Vedant Sanil

1 papers

TL;DR

Abstract

Authors

References96 items

A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

CTRL: Clustering Training Losses for Label Error Detection

Learning from Multiple Annotator Noisy Labels via Sample-wise Label Fusion

Detecting Label Errors by Using Pre-Trained Language Models

Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification

A Survey on Programmatic Weak Supervision

Weak Supervision for Affordable Modeling of Electrocardiogram Data

Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations

Detecting Corrupted Labels Without Training a Model to Predict

Active label cleaning for improved dataset quality under resource constraints

Interactive Label Cleaning with Example-based Explanations

A reconstruction error-based framework for label noise detection

Characterizing The Uncertainty Of Label Noise In Systematic Ultrasound-Guided Prostate Biopsy

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

Understanding deep learning (still) requires rethinking generalization

Early Detection of Sepsis With Machine Learning Techniques: A Brief Clinical Perspective

Clusterability as an Alternative to Anchor Points When Learning with Noisy Labels

Analysing the Noise Model Error for Realistic Noisy Label Data

TabTransformer: Tabular Data Modeling Using Contextual Embeddings

Fair Classification with Group-Dependent Label Noise

TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification

Learning with Instance-Dependent Label Noise: A Sample Sieve Approach

Multiclass classification of dry beans using computer vision and machine learning techniques

Label Noise in Context

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Are we done with ImageNet?

Deep k-NN for Noisy Labels

Label Noise Types and Their Effects on Deep Learning

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

Identifying Mislabeled Data using the Area Under the Margin Ranking

Fantastic Generalization Measures and Where to Find Them

Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels

Meta Label Correction for Noisy Label Learning

Unsupervised Cross-lingual Representation Learning at Scale

Confident Learning: Estimating Uncertainty in Dataset Labels

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Human Uncertainty Makes Classification More Robust

Symmetric Cross Entropy for Robust Learning With Noisy Labels

Are Anchor Points Really Indispensable in Label-Noise Learning?

Combating Label Noise in Deep Learning Using Abstention

How does Disagreement Help Generalization against Label Corruption?

Augmenting the National Institutes of Health Chest Radiograph Dataset with Expert Annotations of Possible Pneumonia.

Learning to Learn From Noisy Labeled Data

The UEA multivariate time series classification archive, 2018

Deep learning for time series classification: a review

Co-teaching: Robust training of deep neural networks with extremely noisy labels

MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

LSTM Fully Convolutional Networks for Time Series Classification

A Closer Look at Memorization in Deep Networks

Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks

Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Robust Loss Functions under Label Noise for Deep Neural Networks

Indexing and classifying gigabytes of time series under time warping

Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs.

The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances

Aggregated Residual Transformations for Deep Neural Networks

Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach

Deep Residual Learning for Image Recognition

Calibrating Probability with Undersampling for Unbalanced Classification

Learning from massive noisy labeled data for image classification

Should We Really Use Post-Hoc Tests Based on Mean-Ranks?

A Rate of Convergence for Mixture Proportion Estimation, with Application to Learning from Noisy Labels

Training Deep Neural Networks on Noisy Labels with Bootstrapping

ImageNet Large Scale Visual Recognition Challenge

Classification in the Presence of Label Noise: A Survey

A Survey of Crowdsourcing Systems

Classification of Household Devices by Electricity Usage Profiles

Learning Word Vectors for Sentiment Analysis

Statistical Comparisons of Classifiers over Multiple Data Sets

Class Noise vs. Attribute Noise: A Quantitative Study

The impact of the MIT-BIH Arrhythmia Database

Combining multiple representations and classifiers for pen-based handwritten digit recognition

Individual Comparisons by Ranking Methods

A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings

Fair Classification with Instance-dependent Label Noise