Normalized Loss Functions for Deep Learning with Noisy Labels

Published in

International Conference on Machine Learning(2020)

External Links:

Generate Graph

TL;DR

Experiments on benchmark datasets demonstrate that the family of new loss functions created by the APL framework can consistently outperform state-of-the-art methods by large margins, especially under large noise rates such as 60% or 80% incorrect labels.

Abstract

Robust loss functions are essential for training accurate deep neural networks (DNNs) in the presence of noisy (incorrect) labels. It has been shown that the commonly used Cross Entropy (CE) loss is not robust to noisy labels. Whilst new loss functions have been designed, they are only partially robust. In this paper, we theoretically show by applying a simple normalization that: any loss can be made robust to noisy labels. However, in practice, simply being robust is not sufficient for a loss function to train accurate DNNs. By investigating several robust loss functions, we find that they suffer from a problem of underfitting. To address this, we propose a framework to build robust loss functions called Active Passive Loss (APL). APL combines two robust loss functions that mutually boost each other. Experiments on benchmark datasets demonstrate that the family of new loss functions created by our APL framework can consistently outperform state-of-the-art methods by large margins, especially under large noise rates such as 60% or 80% incorrect labels.

Authors

S. Erfani

4 papers

Xingjun Ma

7 papers

Hanxun Huang

2 papers

References39 items

SeCoST: Sequential Co-Supervision for Weakly Labeled Audio Event Detection

L_DMI: An Information-theoretic Noise-robust Loss Function

NLNL: Negative Learning for Noisy Labels

Symmetric Cross Entropy for Robust Learning With Noisy Labels

Derivative Manipulation For Adjusting Emphasis Density Function: A General Example Weighting Framework

Normalized Loss Functions for Deep Learning with Noisy Labels

Published in

International Conference on Machine Learning(2020)

External Links:

Generate Graph

TL;DR

Abstract

Authors

S. Erfani

4 papers

Xingjun Ma

7 papers

Hanxun Huang

2 papers

References39 items

SeCoST: Sequential Co-Supervision for Weakly Labeled Audio Event Detection

L_DMI: An Information-theoretic Noise-robust Loss Function

NLNL: Negative Learning for Noisy Labels

Symmetric Cross Entropy for Robust Learning With Noisy Labels

Derivative Manipulation For Adjusting Emphasis Density Function: A General Example Weighting Framework

Simone Romano

1 papers

Derivative Manipulation for General Example Weighting.

IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matters

Improved Mean Absolute Error for Learning Meaningful Patterns from Abnormal Training Data

On Symmetric Losses for Learning from Corrupted Labels

How does Disagreement Help Generalization against Label Corruption?

Dimensionality-Driven Learning with Noisy Labels

Masking: A New Perspective of Noisy Supervision

Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels

Co-teaching: Robust training of deep neural networks with extremely noisy labels

Iterative Learning with Open-set Noisy Labels

Joint Optimization Framework for Learning with Noisy Labels

MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise

WebVision Database: Visual Learning and Understanding from Web Data

Focal Loss for Dense Object Detection

Decoupling "when to update" from "how to update"

Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks

Learning from Noisy Labels with Distillation

Robust Loss Functions under Label Noise for Deep Neural Networks

Regularizing Neural Networks by Penalizing Confident Output Distributions

Learning from Noisy Large-Scale Datasets with Minimal Supervision

Training deep neural-networks using a noise adaptation layer

Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Learning from massive noisy labeled data for image classification

Training Deep Neural Networks on Noisy Labels with Bootstrapping

Training Convolutional Networks with Noisy Labels

Learning with Noisy Labels

ImageNet: A large-scale hierarchical image database

Making Neural Networks Robust to Label Noise: a Loss Correction Approach

Learning Multiple Layers of Features from Tiny Images

GradientBased Learning Applied to Document Recognition

Gradient-based learning applied to document recognition

Field of Study

Computer ScienceMathematics

Venue Information

Name

International Conference on Machine Learning

Type

conference

URL

https://icml.cc/

Alternate Names

ICML
Int Conf Mach Learn