Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020-06-13T00:00:00.000000Z)

TL;DR

This work introduces Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning that performs on par or better than the current state of the art on both transfer and semi- supervised benchmarks.

Abstract

We introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.

Authors

K. Kavukcuoglu

26 papers

Bilal Piot

6 papers

M. G. Azar

4 papers

TL;DR

Abstract

Authors

References103 items

ScaleNet: An Unsupervised Representation Learning Method for Limited Information

Big Self-Supervised Models are Strong Semi-Supervised Learners

What makes for good views for contrastive learning

Prototypical Contrastive Learning of Unsupervised Representations

Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning

ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring

Improved Baselines with Momentum Contrastive Learning

MaxUp: A Simple Way to Improve Generalization of Neural Network Training

A Simple Framework for Contrastive Learning of Visual Representations

FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

Self-Supervised Learning of Pretext-Invariant Representations

Momentum Contrast for Unsupervised Visual Representation Learning

Semi-Supervised Learning

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark

Randaugment: Practical automated data augmentation with a reduced search space

Large Scale Adversarial Representation Learning

Contrastive Multiview Coding

Learning Representations by Maximizing Mutual Information Across Views

Data-Efficient Image Recognition with Contrastive Predictive Coding

On Variational Bounds of Mutual Information

S4L: Self-Supervised Semi-Supervised Learning

MixMatch: A Holistic Approach to Semi-Supervised Learning

Local Aggregation for Unsupervised Learning of Visual Embeddings

A Theoretical Analysis of Contrastive Unsupervised Representation Learning

Revisiting Self-Supervised Visual Representation Learning

Deep Reinforcement Learning and the Deadly Triad

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Learning deep representations by mutual information estimation and maximization

Representation Learning with Contrastive Predictive Coding

Places: A 10 Million Image Database for Scene Recognition

Do Better ImageNet Models Transfer Better?

Unsupervised Feature Learning via Non-parametric Instance Discrimination

Group Normalization

Unsupervised Representation Learning by Predicting Image Rotations

Rainbow: Combining Improvements in Deep Reinforcement Learning

Multi-task Self-Supervised Visual Learning

Scaling SGD Batch Size to 32K for ImageNet Training

Sampling Matters in Deep Embedding Learning

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

Smart Mining for Deep Metric Learning

Weight-averaged consistency targets improve semi-supervised deep learning results

Learning Features by Watching Objects Move

Temporal Ensembling for Semi-Supervised Learning

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

SGDR: Stochastic Gradient Descent with Warm Restarts

Adversarially Learned Inference

Deeper Depth Prediction with Fully Convolutional Residual Networks

Adversarial Feature Learning

Wide Residual Networks

Context Encoders: Feature Learning by Inpainting

Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

Colorful Image Colorization

Learning Representations for Automatic Colorization

Asynchronous Methods for Deep Reinforcement Learning

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Continuous control with deep reinforcement learning

Semi-supervised Learning with Ladder Networks

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Unsupervised Visual Representation Learning by Context Prediction

Human-level control through deep reinforcement learning

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Learning with Pseudo-Ensembles

Fully convolutional networks for semantic segmentation

Food-101 - Mining Discriminative Components with Random Forests

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Discriminative Unsupervised Feature Learning with Convolutional Neural Networks

Birdsnap: Large-Scale Fine-Grained Visual Categorization of Birds

Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks

Semi-supervised Learning with Deep Generative Models

Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes