In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning (2021-01-15T00:00:00.000000Z)

TL;DR

This work proposes an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process and generalizes the pseudo- labeling process, allowing for the creation of negative pseudo-labels.

Abstract

The recent research in semi-supervised learning (SSL) is mostly dominated by consistency regularization based methods which achieve strong performance. However, they heavily rely on domain-specific data augmentations, which are not easy to generate for all data modalities. Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models; these predictions generate many incorrect pseudo-labels, leading to noisy training. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process. Furthermore, UPS generalizes the pseudo-labeling process, allowing for the creation of negative pseudo-labels; these negative pseudo-labels can be used for multi-label classification as well as negative learning to improve the single-label classification. We achieve strong performance when compared to recent SSL methods on the CIFAR-10 and CIFAR-100 datasets. Also, we demonstrate the versatility of our method on the video dataset UCF-101 and the multi-label dataset Pascal VOC.

Authors

M. Shah

20 papers

Y. Rawat

4 papers

Mamshad Nayeem Rizve

3 papers

TL;DR

Abstract

Authors

References79 items

Uncertainty-aware Self-training for Few-shot Text Classification

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Big Self-Supervised Models are Strong Semi-Supervised Learners

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring

Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation

VideoSSL: Semi-Supervised Learning for Video Classification

A Simple Framework for Contrastive Learning of Visual Representations

Iterative Label Improvement: Robust Training by Confidence Based Filtering and Dataset Partitioning

FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

Distance-Based Learning from Errors for Confidence Calibration

Self-Training With Noisy Student Improves ImageNet Classification

RandAugment: Practical data augmentation with no separate search

NLNL: Negative Learning for Noisy Labels

Repetitive Reprediction Deep Decipher for Semi-Supervised Learning

Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation

On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks

S4L: Self-Supervised Semi-Supervised Learning

MixMatch: A Holistic Approach to Semi-Supervised Learning

Label Propagation for Deep Semi-Supervised Learning

Probabilistic End-To-End Noise Correction for Learning With Noisy Labels

Interpolation Consistency Training for Semi-Supervised Learning

A Simple Baseline for Bayesian Uncertainty in Deep Learning

Deep Metric Transfer for Label Propagation with Limited Annotated Data

3D Semi-Supervised Learning with Uncertainty-Aware Multi-View Co-Training

A Baseline for Multi-Label Image Classification Using an Ensemble of Deep Convolutional Neural Networks

DropBlock: A regularization method for convolutional networks

Transductive Semi-Supervised Deep Learning Using Min-Max Features

Predictive Uncertainty Estimation via Prior Networks

Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?

Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning

mixup: Beyond Empirical Risk Minimization

Improved Regularization of Convolutional Neural Networks with Cutout

On Calibration of Modern Neural Networks

Shake-Shake regularization

Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

Prototypical Networks for Few-shot Learning

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Temporal Ensembling for Semi-Supervised Learning

Variational Autoencoder for Deep Learning of Images, Labels and Captions

SGDR: Stochastic Gradient Descent with Warm Restarts

Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning

Matching Networks for One Shot Learning

Wide Residual Networks

Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors

Deep Residual Learning for Image Recognition

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Weight Uncertainty in Neural Networks

Unsupervised Visual Representation Learning by Context Prediction

Efficient object localization using Convolutional Networks

Semi-supervised Learning with Deep Generative Models

Regularization of Neural Networks using DropConnect

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Practical Variational Inference for Neural Networks

Bayesian Learning via Stochastic Gradient Langevin Dynamics

Principled Hybrids of Generative and Discriminative Models

Effective Self-Training for Parsing

Semi-supervised Learning by Entropy Minimization

Transductive Learning via Spectral Graph Partitioning

Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering

Transductive Inference for Text Classification using Support Vector Machines

Learning by Transduction

Unsupervised Word Sense Disambiguation Rivaling Supervised Methods

The Comparison and Evaluation of Forecasters.

The Well-Calibrated Bayesian

Time-Consistent Self-Supervision for Semi-Supervised Learning

Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning

Realistic Evaluation of Semi-Supervised Learning Algorithms

Semi-Supervised Learning

Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks

Learning Multiple Layers of Features from Tiny Images

VOC2007 and UCF-101 datasets the fully supervised classification scores

The PASCAL Visual Object Classes Challenge