Masked Autoencoders Are Scalable Vision Learners (2021-11-11T00:00:00.000000Z)

TL;DR

This paper develops an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens.

Abstract

This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3× or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior.

Authors

Ross B. Girshick

23 papers

Saining Xie

10 papers

Kaiming He

3 papers

TL;DR

Abstract

Authors

References73 items

Benchmarking Detection Transfer Learning with Vision Transformers

Early Convolutions Help Transformers See Better

VOLO: Vision Outlooker for Visual Recognition

BEiT: BERT Pre-Training of Image Transformers

Quality-Agnostic Image Recognition via Invertible Decoder

Towards Robust Vision Transformer

Emerging Properties in Self-Supervised Vision Transformers

An Empirical Study of Training Self-Supervised Vision Transformers

Self-supervised Pretraining of Visual Features in the Wild

Zero-Shot Text-to-Image Generation

Training data-efficient image transformers & distillation through attention

Grafit: Learning fine-grained image representations with coarse labels

Exploring Simple Siamese Representation Learning

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Generative Pretraining From Pixels

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Language Models are Few-Shot Learners

A Simple Framework for Contrastive Learning of Visual Representations

Momentum Contrast for Unsupervised Visual Representation Learning

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Randaugment: Practical automated data augmentation with a reduced search space

Natural Adversarial Examples

Fixing the train-test resolution discrepancy

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Learning Robust Global Representations by Penalizing Local Predictive Power

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Unified Perceptual Parsing for Scene Understanding

Representation Learning with Contrastive Predictive Coding

Unsupervised Feature Learning via Non-parametric Instance Discrimination

Exploring the Limits of Weakly Supervised Pretraining

Unsupervised Representation Learning by Predicting Image Rotations

Decoupled Weight Decay Regularization

Neural Discrete Representation Learning

mixup: Beyond Empirical Risk Minimization

Large Batch Training of Convolutional Networks

The iNaturalist Species Classification and Detection Dataset

Attention is All you Need

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Mask R-CNN

Learning Features by Watching Objects Move

Feature Pyramid Networks for Object Detection

Semantic Understanding of Scenes Through the ADE20K Dataset

SGDR: Stochastic Gradient Descent with Warm Restarts

Layer Normalization

Context Encoders: Feature Learning by Inpainting

Deep Networks with Stochastic Depth

Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

Colorful Image Colorization

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Unsupervised Visual Representation Learning by Context Prediction

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Learning Deep Features for Scene Recognition using Places Database

How transferable are features in deep neural networks?

Very Deep Convolutional Networks for Large-Scale Image Recognition

Microsoft COCO: Common Objects in Context

ImageNet classification with deep convolutional neural networks

Understanding the difficulty of training deep feedforward neural networks

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion

ImageNet: A large-scale hierarchical image database

Extracting and composing robust features with denoising autoencoders

Dimensionality Reduction by Learning an Invariant Mapping

Support-Vector Networks

Autoencoders, Minimum Description Length and Helmholtz Free Energy

Self-organizing neural network that discovers surfaces in random-dot stereograms

Backpropagation Applied to Handwritten Zip Code Recognition

ELECTRA: Pre-training text encoders as discriminators rather than generators

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Improving Language Understanding by Generative Pre-Training

The iNaturalist Species Classification and Detection Dataset-Supplementary Material

Unsupervised Learning of Visual Representations using Videos

Field of Study

Journal Information

Name