Self-Supervision Can Be a Good Few-Shot Learner (2022-07-19T00:00:00.000000Z)

TL;DR

This work proposes an effective unsupervised FSL method, learning representations with self-supervision, following the InfoMax principle, which achieves comparable performance on widely used FSL benchmarks without any labels of the base classes.

Abstract

Existing few-shot learning (FSL) methods rely on training with a large labeled dataset, which prevents them from leveraging abundant unlabeled data. From an information-theoretic perspective, we propose an effective unsupervised FSL method, learning representations with self-supervision. Following the InfoMax principle, our method learns comprehensive representations by capturing the intrinsic structure of the data. Specifically, we maximize the mutual information (MI) of instances and their representations with a low-bias MI estimator to perform self-supervised pre-training. Rather than supervised pre-training focusing on the discriminable features of the seen classes, our self-supervised model has less bias toward the seen classes, resulting in better generalization for unseen classes. We explain that supervised pre-training and self-supervised pre-training are actually maximizing different MI objectives. Extensive experiments are further conducted to analyze their FSL performance with various training settings. Surprisingly, the results show that self-supervised pre-training can outperform supervised pre-training under the appropriate conditions. Compared with state-of-the-art FSL methods, our approach achieves comparable performance on widely used FSL benchmarks without any labels of the base classes.

Authors

Xinmei Tian

2 papers

Jianzhuang Liu

4 papers

Yuning Lu

1 papers

TL;DR

Abstract

Authors

References90 items

How Well Do Self-Supervised Models Transfer to Medical Imaging?

Prompt Distribution Learning

Understanding Dimensional Collapse in Contrastive Self-supervised Learning

Binocular Mutual Learning for Improving Few-shot Classification

Pareto Self-Supervised Training for Few-Shot Learning

Self-Supervised Training Enhances Online Continual Learning

Barlow Twins: Self-Supervised Learning via Redundancy Reduction

Learning Transferable Visual Models From Natural Language Supervision

Self-Supervised Features Improve Open-World Learning

Understanding self-supervised Learning Dynamics without Contrastive Pairs

Free Lunch for Few-shot Learning: Distribution Calibration

On Episodes, Prototypical Networks, and Few-shot Learning

How Well Do Self-Supervised Models Transfer?

Exploring Simple Siamese Representation Learning

Intriguing Properties of Contrastive Losses

Self-training for Few-shot Transfer Across Extreme Task Differences

Shot in the Dark: Few-Shot Learning with No Base-Class Labels

Transductive Information Maximization For Few-Shot Learning

Prevalence of neural collapse during the terminal phase of deep learning training

Self-Supervised Prototypical Transfer Learning for Few-Shot Classification

Unsupervised Meta-Learning through Latent-Space Interpolation in Generative Models

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Language Models are Few-Shot Learners

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

Mutual Information Gradient Estimation for Representation Learning

Unsupervised Few-shot Learning via Distribution Shift-based Augmentation

Adversarial Feature Hallucination Networks for Few-Shot Learning

Negative Margin Matters: Understanding Margin in Few-shot Classification

Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?

A Unifying Mutual Information View of Metric Learning: Cross-Entropy vs. Pairwise Losses

Improved Baselines with Momentum Contrastive Learning

A New Meta-Baseline for Few-Shot Learning

A Simple Framework for Contrastive Learning of Visual Representations

A Broader Study of Cross-Domain Few-Shot Learning

Associative Alignment for Few-shot Image Classification

Momentum Contrast for Unsupervised Visual Representation Learning

Contrastive Representation Distillation

Understanding the Limitations of Variational Mutual Information Estimators

When Does Self-supervision Improve Few-shot Learning?

Randaugment: Practical automated data augmentation with a reduced search space

A Baseline for Few-Shot Image Classification

Meta-Learning with Warped Gradient Descent

On Mutual Information Maximization for Representation Learning

Charting the Right Manifold: Manifold Mixup for Few-shot Learning

Contrastive Multiview Coding

Boosting Few-Shot Visual Learning With Self-Supervision

Image Deformation Meta-Networks for One-Shot Learning

On Variational Bounds of Mutual Information

A Closer Look at Few-shot Classification

Meta-Learning With Differentiable Convex Optimization

Diversity With Cooperation: Ensemble Methods for Few-Shot Classification

Assume, Augment and Learn: Unsupervised Few-Shot Meta-Learning via Random Labels and Data Augmentation

Few-Shot Learning via Embedding Adaptation With Set-to-Set Functions

Unsupervised Meta-Learning for Few-Shot Image Classification

Unsupervised Learning via Meta-Learning

Learning deep representations by mutual information estimation and maximization

Improving Generalization via Scalable Neighborhood Component Analysis

Meta-Learning with Latent Embedding Optimization

Representation Learning with Contrastive Predictive Coding

Delta-encoder: an effective sample synthesis method for few-shot object recognition

TADAM: Task dependent adaptive metric for improved few-shot learning

Unsupervised Feature Learning via Non-parametric Instance Discrimination

Meta-Learning for Semi-Supervised Few-Shot Classification

Unsupervised Representation Learning by Predicting Image Rotations

Mutual Information Neural Estimation

Learning to Compare: Relation Network for Few-Shot Learning

A Simple Neural Attentive Meta-Learner

Prototypical Networks for Few-shot Learning

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Optimization as a Model for Few-Shot Learning

Matching Networks for One Shot Learning

Context Encoders: Feature Learning by Inpainting

Colorful Image Colorization

Learning Representations for Automatic Colorization

Human-level concept learning through probabilistic program induction

Deep Residual Learning for Image Recognition