Unsupervised Selective Labeling for More Effective Semi-supervised Learning (2021-10-06T00:00:00.000000Z)

TL;DR

This work sets a new standard for practical and efficient semi-supervised learning on partially labeled data, and consistently improves SSL methods over state-of-the-art active learning given labeled data by 8 to 25 times in label efficiency.

Abstract

Given an unlabeled dataset and an annotation budget, we study how to selectively label a fixed number of instances so that semi-supervised learning (SSL) on such a partially labeled dataset is most effective. We focus on selecting the right data to label, in addition to usual SSL's propagating labels from labeled data to the rest unlabeled data. This instance selection task is challenging, as without any labeled data we do not know what the objective of learning should be. Intuitively, no matter what the downstream task is, instances to be labeled must be representative and diverse: The former would facilitate label propagation to unlabeled data, whereas the latter would ensure coverage of the entire dataset. We capture this idea by selecting cluster prototypes, either in a pretrained feature space, or along with feature optimization, both without labels. Our unsupervised selective labeling consistently improves SSL methods over state-of-the-art active learning given labeled data, by 8 to 25 times in label efficiency. For example, it boosts FixMatch by 10% (14%) in accuracy on CIFAR-10 (ImageNet-1K) with 0.08% (0.2%) labeled data, demonstrating that small computation spent on selecting what data to label brings significant gain especially under a low annotation budget. Our work sets a new standard for practical and efficient SSL.

Authors

Xudong Wang

4 papers

Long Lian

2 papers

Stella X. Yu

1 papers

TL;DR

Abstract

Authors

References87 items

ScaleNet: An Unsupervised Representation Learning Method for Limited Information

Debiased Learning from Naturally Imbalanced Pseudo-Labels

Active Learning at the ImageNet Scale

MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification

Semi-supervised Active Learning for Semi-supervised Models: Exploit Adversarial Examples with Graph-based Virtual Labels

Semi-Supervised Active Learning with Temporal Output Discrepancy

Improve Unsupervised Pretraining for Few-label Transfer

MCDAL: Maximum Classifier Discrepancy for Active Learning

Nearest Neighbor Matching for Deep Clustering

Iterative human and automated identification of wildlife images

Emerging Properties in Self-Supervised Vision Transformers

Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples

Learning Transferable Visual Models From Natural Language Supervision

Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning

Improving Unsupervised Image Clustering With Robust Learning

CoMatch: Semi-supervised Learning with Contrastive Graph Regularization

On the Marginal Benefit of Active Learning: Does Self-Supervision Eat its Cake?

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

A Survey of Deep Active Learning

Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination

Long-tail learning via logit adjustment

Unsupervised Image Classification for Deep Representation Learning

Sequential Graph Convolutional Network for Active Learning

Big Self-Supervised Models are Strong Semi-Supervised Learners

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Learning To Classify Images Without Labels

Improved Baselines with Momentum Contrastive Learning

A Simple Framework for Contrastive Learning of Visual Representations

FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring

Momentum Contrast for Unsupervised Visual Representation Learning

Self-Training With Noisy Student Improves ImageNet Classification

Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications

Consistency-Based Semi-Supervised Active Learning: Towards Minimizing Labeling Cost

Randaugment: Practical automated data augmentation with a reduced search space

Combining MixMatch and Active Learning for Better Accuracy with Fewer Labels

Learning Loss for Active Learning

MixMatch: A Holistic Approach to Semi-Supervised Learning

Unsupervised Data Augmentation for Consistency Training

Variational Adversarial Active Learning

Local Aggregation for Unsupervised Learning of Visual Embeddings

Spectral graph theory

Deep Metric Transfer for Label Propagation with Limited Annotated Data

Representation Learning with Contrastive Predictive Coding

Deep k-Means: Jointly Clustering with k-Means and Learning Representations

Unsupervised Feature Learning via Non-parametric Instance Discrimination

Adversarial Active Learning for Deep Networks: a Margin Based Approach

Deep Learning Scaling is Predictable, Empirically

Large Batch Training of Convolutional Networks

Active Learning for Convolutional Neural Networks: A Core-Set Approach

Weight-averaged consistency targets improve semi-supervised deep learning results

The (black) art of runtime evaluation: Are we comparing algorithms or implementations?

Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning

Deep Residual Learning for Image Recognition

Unsupervised Deep Embedding for Clustering Analysis

Submodularity in Data Subset Selection and Active Learning

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

ImageNet Large Scale Visual Recognition Challenge

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

ImageNet classification with deep convolutional neural networks

K-nearest neighbour kernel density estimation, the choice of optimal k

Natural Language Processing (Almost) from Scratch

Active Learning by Querying Informative and Representative Examples

Image Clustering Using Local Discriminant Models and Global Integration

Unsupervised Bayesian Convex Deconvolution Based on a Field With an Explicit Partition Function

Dimensionality Reduction by Learning an Invariant Mapping

Semi-supervised Learning by Entropy Minimization

Toward Optimal Active Learning through Sampling Estimation of Error Reduction

Support Vector Machine Active Learning with Applications to Text Classification

Normalized cuts and image segmentation

Improving Generalization with Active Learning

Discriminatory Analysis - Nonparametric Discrimination: Consistency Properties

Least squares quantization in PCM

Graph theory with applications to engineering and computer science

Deep Learning