Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach (2016-09-13T00:00:00.000000Z)

TL;DR

It is proved that, when ReLU is the only non-linearity, the loss curvature is immune to class-dependent label noise, and it is shown how one can estimate these probabilities, adapting a recent technique for noise estimation to the multi-class setting, and providing an end-to-end framework.

Abstract

We present a theoretically grounded approach to train deep neural networks, including recurrent networks, subject to class-dependent label noise. We propose two procedures for loss correction that are agnostic to both application domain and network architecture. They simply amount to at most a matrix inversion and multiplication, provided that we know the probability of each class being corrupted into another. We further show how one can estimate these probabilities, adapting a recent technique for noise estimation to the multi-class setting, and thus providing an end-to-end framework. Extensive experiments on MNIST, IMDB, CIFAR-10, CIFAR-100 and a large scale dataset of clothing images employing a diversity of architectures — stacking dense, convolutional, pooling, dropout, batch normalization, word embedding, LSTM and residual layers — demonstrate the noise robustness of our proposals. Incidentally, we also prove that, when ReLU is the only non-linearity, the loss curvature is immune to class-dependent label noise.

Authors

A. Menon

4 papers

Giorgio Patrini

2 papers

A. Rozza

1 papers

TL;DR

Abstract

Authors

References45 items

Deep Learning without Poor Local Minima

Learning from Binary Labels with Instance-Dependent Corruption

Deep Networks with Stochastic Depth

Identity Mappings in Deep Residual Networks

Mixture Proportion Estimation via Kernel Embeddings of Distributions

Loss factorization, weakly supervised learning and label noise robustness

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Deep Residual Learning for Image Recognition

Efficient and Robust Automated Machine Learning

The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition

Semi-supervised Sequence Learning

Machine learning via transitions

Learning from Corrupted Binary Labels via Class-Probability Estimation

Learning from massive noisy labeled data for image classification

Visual recognition by learning from web data: A weakly supervised domain generalization approach

Learning with Symmetric Label Noise: The Importance of Being Unhinged

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Training Deep Neural Networks on Noisy Labels with Bootstrapping

Learning Everything about Anything: Webly-Supervised Visual Concept Learning

Learning from Noisy Labels with Deep Neural Networks

Classification in the Presence of Label Noise: A Survey

Making risk minimization tolerant to label noise

Learning with Noisy Labels

Class Proportion Estimation with Application to Multiclass Anomaly Rejection

Classification with Asymmetric Label Noise: Consistency and Maximal Denoising

ImageNet classification with deep convolutional neural networks

Learning to Label Aerial Images from Noisy Data

Learning Word Vectors for Sentiment Analysis

Harvesting Image Databases from the Web

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

Learning Object Categories From Internet Image Searches

Understanding the difficulty of training deep feedforward neural networks

Composite Binary Losses

Learning SVMs from Sloppily Labeled Data

On the Design of Loss Functions for Classification: theory, robustness to outliers, and SavageBoost

Random classification noise defeats all convex potential boosters

Harvesting Image Databases from the Web

Long Short-Term Memory

Efﬁcient and Robust Automated Machine Learning

Dropout: a simple way to prevent neural networks from overfitting

Learning Multiple Layers of Features from Tiny Images

GradientBased Learning Applied to Document Recognition

Gradient-based learning applied to document recognition

Ieee Transactions on Pattern Analysis and Machine Intelligence Classification with Noisy Labels by Importance Reweighting

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names