An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020-10-22T00:00:00.000000Z)

TL;DR

Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

Abstract

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

Authors

N. Houlsby

12 papers

Xiaohua Zhai

15 papers

Thomas Unterthiner

8 papers

TL;DR

Abstract

Authors

References65 items

On Robustness and Transferability of Convolutional Neural Networks

Generative Pretraining From Pixels

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Object-Centric Learning with Slot Attention

Are we done with ImageNet?

Visual Transformers: Token-based Image Representation and Processing for Computer Vision

Language Models are Few-Shot Learners

End-to-End Object Detection with Transformers

Quantifying Attention Flow in Transformers

Exploring Self-Attention for Image Recognition

Fixing the train-test resolution discrepancy: FixEfficientNet

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

A Simple Framework for Contrastive Learning of Visual Representations

Big Transfer (BiT): General Visual Representation Learning

Self-Supervised Learning of Video-Induced Visual Invariances

Momentum Contrast for Unsupervised Visual Representation Learning

Self-Training With Noisy Student Improves ImageNet Classification

On the Relationship between Self-Attention and Convolutional Layers

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark

UNITER: UNiversal Image-TExt Representation Learning

Axial Attention in Multidimensional Transformers

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Fixing the train-test resolution discrepancy

Stand-Alone Self-Attention in Vision Models

Scaling Autoregressive Video Models

Learning Deep Transformer Models for Machine Translation

Learning Representations by Maximizing Mutual Information Across Views

Data-Efficient Image Recognition with Contrastive Predictive Coding

S4L: Self-Supervised Semi-Supervised Learning

Local Relation Networks for Image Recognition

Generating Long Sequences with Sparse Transformers

Attention Augmented Convolutional Networks

VideoBERT: A Joint Model for Video and Language Representation Learning

Weight Standardization

CCNet: Criss-Cross Attention for Semantic Segmentation

Adaptive Input Representations for Neural Language Modeling

Exploring the Limits of Weakly Supervised Pretraining

Group Normalization

Image Transformer

Relation Networks for Object Detection

Non-local Neural Networks

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Attention is All you Need

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

Deep Residual Learning for Image Recognition

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

ImageNet classification with deep convolutional neural networks

ImageNet: A large-scale hierarchical image database

Automated Flower Classification over a Large Number of Classes

Acceleration of stochastic approximation by averaging

Backpropagation Applied to Handwritten Zip Code Recognition

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Improving language understanding with unsupervised learning

Learning Multiple Layers of Features from Tiny Images

cats and dogs

ViT com putationally m ore efficient

Training can benefit from exis ting optim izations (hardware) for trans form ers

Split im age into patches (like ViT) followed by a linear trans form ation

Scope for im provem ent with m ore data/ param eters

Trans form ers can work for im ages . Es pecially when data is large

Merge patches (2x2 neighbors )

Under review as a conference paper at ICLR 2019 softmax Transpose Updated Memory Memory embedded input linear concat Attention Weights concat

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names