Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations (2021-10-22T00:00:00.000000Z)

TL;DR

It is shown that the real-world noise patterns impose new and outstanding challenges as compared to synthetic label noise, and the availability of these two datasets would facilitate the development and evaluation of future learning with noisy label solutions.

Abstract

Existing research on learning with noisy labels mainly focuses on synthetic label noise. Synthetic noise, though has clean structures which greatly enabled statistical analyses, often fails to model real-world noise patterns. The recent literature has observed several efforts to offer real-world noisy datasets, yet the existing efforts suffer from two caveats: (1) The lack of ground-truth verification makes it hard to theoretically study the property and treatment of real-world label noise; (2) These efforts are often of large scales, which may result in unfair comparisons of robust methods within reasonable and accessible computation power. To better understand real-world label noise, it is crucial to build controllable and moderate-sized real-world noisy datasets with both ground-truth and noisy labels. This work presents two new benchmark datasets CIFAR-10N, CIFAR-100N, equipping the training datasets of CIFAR-10, CIFAR-100 with human-annotated real-world noisy labels we collected from Amazon Mechanical Turk. We quantitatively and qualitatively show that real-world noisy labels follow an instance-dependent pattern rather than the classically assumed and adopted ones (e.g., class-dependent label noise). We then initiate an effort to benchmarking a subset of the existing solutions using CIFAR-10N and CIFAR-100N. We further proceed to study the memorization of correct and wrong predictions, which further illustrates the difference between human noise and class-dependent synthetic noise. We show indeed the real-world noise patterns impose new and outstanding challenges as compared to synthetic label noise. These observations require us to rethink the treatment of noisy labels, and we hope the availability of these two datasets would facilitate the development and evaluation of future learning with noisy label solutions. Datasets and leaderboards are available at http://noisylabels.com.

References56 items

Robust Training under Label Noise by Over-parameterization

The Rich Get Richer: Disparate Impact of Semi-Supervised Learning

Understanding and Improving Early Stopping for Learning with Noisy Labels

Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets

Learning with Feature-Dependent Label Noise: A Progressive Approach

Understanding deep learning (still) requires rethinking generalization

Clusterability as an Alternative to Anchor Points When Learning with Noisy Labels

Provably End-to-end Label-Noise Learning without Anchor Points

A Second-Order Approach to Learning with Instance-Dependent Label Noise

Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting

A Survey of Label-noise Representation Learning: Past, Present and Future

When Optimizing f-divergence is Robust with Label Noise

Fair Classification with Group-Dependent Label Noise

Policy Learning Using Weak Supervision

Learning with Instance-Dependent Label Noise: A Sample Sieve Approach

Early-Learning Regularization Prevents Memorization of Noisy Labels

Parts-dependent Label Noise: Towards Instance-dependent Label Noise

Dual T: Reducing Estimation Error for Transition Matrix in Label-noise Learning

Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization

Does label smoothing mitigate label noise?

DivideMix: Learning with Noisy Labels as Semi-supervised Learning

Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels

Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates

Human Uncertainty Makes Classification More Robust

Symmetric Cross Entropy for Robust Learning With Noisy Labels

Robust Bi-Tempered Logistic Loss Based on Bregman Divergences

Are Anchor Points Really Indispensable in Label-Noise Learning?

SELFIE: Refurbishing Unclean Samples for Robust Deep Learning

MixMatch: A Holistic Approach to Semi-Supervised Learning

How does Disagreement Help Generalization against Label Corruption?

Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels

Co-teaching: Robust training of deep neural networks with extremely noisy labels

Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise

MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise

WebVision Database: Visual Learning and Understanding from Web Data

A Closer Look at Memorization in Deep Networks

Robust Loss Functions under Label Noise for Deep Neural Networks

Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach

Matching Networks for One Shot Learning

Deep Residual Learning for Image Recognition

Multi-class Multi-annotator Active Learning with Robust Gaussian Process for Visual Recognition

Learning from massive noisy labeled data for image classification

Food-101 - Mining Discriminative Components with Random Forests

Learning with Noisy Labels

3D Object Representations for Fine-Grained Categorization

Collaborative Active Learning of a Kernel Machine Ensemble for Recognition

An Information Fusion Approach to Learning with Instance-Dependent Label Noise

Robust early-learning: Hindering the memorization of noisy labels

A Good Representation Detects Noisy Labels

Understanding (Generalized) Label Smoothing when Learning with Noisy Labels

2020] and DivideMix since either advanced data augmentation strategies or two deep neural networks

SELFIE: Refurbishing Unclean Samples for Robust Deep Learning

Learning Multiple Layers of Features from Tiny Images

Ieee Transactions on Pattern Analysis and Machine Intelligence Classification with Noisy Labels by Importance Reweighting

COMPUTING INFRASTRUCTURE All our experiments run on a GPU cluster (500 GPUs of all kinds, mainly use 2080 Ti) for training and evaluation