BEiT: BERT Pre-Training of Image Transformers (2021-06-15T00:00:00.000000Z)

TL;DR

A self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers, is introduced, and results on image classification and semantic segmentation show that the model achieves competitive results with previous pre-training methods.

Abstract

We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first"tokenize"the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder. Experimental results on image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%). The code and pretrained models are available at https://aka.ms/beit.

Authors

Li Dong

11 papers

Furu Wei

37 papers

Hangbo Bao

3 papers

TL;DR

Abstract

Authors

References59 items

Scaling Vision Transformers

Efficient Training of Visual Transformers with Small Datasets

Self-Supervised Learning with Swin Transformers

Emerging Properties in Self-Supervised Vision Transformers

mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

An Empirical Study of Training Self-Supervised Vision Transformers

Going deeper with Image Transformers

Zero-Shot Text-to-Image Generation

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Training data-efficient image transformers & distillation through attention

Exploring Simple Siamese Representation Learning

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Generative Pretraining From Pixels

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Prototypical Contrastive Learning of Unsupervised Representations

Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

Improved Baselines with Momentum Contrastive Learning

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

A Simple Framework for Contrastive Learning of Visual Representations

Momentum Contrast for Unsupervised Visual Representation Learning

Self-labelling via simultaneous clustering and representation learning

Unsupervised Cross-lingual Representation Learning at Scale

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

RoBERTa: A Robustly Optimized BERT Pretraining Approach

SpanBERT: Improving Pre-training by Representing and Predicting Spans

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Selfie: Self-supervised Pretraining for Image Embedding

Learning Representations by Maximizing Mutual Information Across Views

Generating Diverse High-Fidelity Images with VQ-VAE-2

Unified Language Model Pre-training for Natural Language Understanding and Generation

Cross-lingual Language Model Pretraining

Learning deep representations by mutual information estimation and maximization

Unified Perceptual Parsing for Scene Understanding

Deep Clustering for Unsupervised Learning of Visual Features

Representation Learning with Contrastive Predictive Coding

Unsupervised Feature Learning via Non-parametric Instance Discrimination

Unsupervised Representation Learning by Predicting Image Rotations

Decoupled Weight Decay Regularization

Neural Discrete Representation Learning

Attention is All you Need

Categorical Reparameterization with Gumbel-Softmax

The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables

Semantic Understanding of Scenes Through the ADE20K Dataset

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Deep Networks with Stochastic Depth

Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

Colorful Image Colorization

Neural Machine Translation of Rare Words with Subword Units

Adam: A Method for Stochastic Optimization

ImageNet Large Scale Visual Recognition Challenge

Microsoft COCO: Common Objects in Context

Auto-Encoding Variational Bayes

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Learning Multiple Layers of Features from Tiny Images

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names