Identifying Incorrect Annotations in Multi-Label Classification Data (2022-11-25T00:00:00.000000Z)

TL;DR

An extension of the Confident Learning framework is proposed to this setting, as well as a label quality score that ranks examples with label errors much higher than those which are correctly labeled.

Abstract

In multi-label classification, each example in a dataset may be annotated as belonging to one or more classes (or none of the classes). Example applications include image (or document) tagging where each possible tag either applies to a particular image (or document) or not. With many possible classes to consider, data annotators are likely to make errors when labeling such data in practice. Here we consider algorithms for finding mislabeled examples in multi-label classification datasets. We propose an extension of the Confident Learning framework to this setting, as well as a label quality score that ranks examples with label errors much higher than those which are correctly labeled. Both approaches can utilize any trained classifier. After demonstrating that our methodology empirically outperforms other algorithms for label error detection, we apply our approach to discover many label errors in the CelebA image tagging dataset.

Authors

Curtis G. Northcutt

4 papers

Jonas W. Mueller

6 papers

Aditya Thyagarajan

1 papers

Identifying Incorrect Annotations in Multi-Label Classification Data

TL;DR

Abstract

Authors

References19 items

Detecting Label Errors in Token Classification Data

Automated Detection of Label Errors in Semantic Segmentation Datasets via Deep Learning and Uncertainty Quantification

Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

Identifying Incorrect Labels in the CoNLL-2003 Corpus

Learning From Noisy Labels With Deep Neural Networks: A Survey

Confident Learning: Estimating Uncertainty in Dataset Labels

O2U-Net: A Simple Noisy Label Detection Approach for Deep Neural Networks

Identifying Mislabeled Instances in Classification Datasets

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise

Adam: A Method for Stochastic Optimization

Deep Learning Face Attributes in the Wild

Learning with Noisy Labels

Scikit-learn: Machine Learning in Python

Identifying Mislabeled Training Data

Data-centric ai competition, 2021

Pytorch image models

Model-Agnostic Label Quality Scoring to Detect Real-World Label Errors

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names