DaViT: Dual Attention Vision Transformers

Published in

European Conference on Computer Vision(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

This work introduces Dual Attention Vision Transformers (DaViT), a simple yet effective vision transformer architecture that is able to capture global context while maintaining computational efficiency and achieves state-of-the-art performance on four different tasks with efficient computations.

Abstract

In this work, we introduce Dual Attention Vision Transformers (DaViT), a simple yet effective vision transformer architecture that is able to capture global context while maintaining computational efficiency. We propose approaching the problem from an orthogonal angle: exploiting self-attention mechanisms with both"spatial tokens"and"channel tokens". With spatial tokens, the spatial dimension defines the token scope, and the channel dimension defines the token feature dimension. With channel tokens, we have the inverse: the channel dimension defines the token scope, and the spatial dimension defines the token feature dimension. We further group tokens along the sequence direction for both spatial and channel tokens to maintain the linear complexity of the entire model. We show that these two self-attentions complement each other: (i) since each channel token contains an abstract representation of the entire image, the channel attention naturally captures global interactions and representations by taking all spatial positions into account when computing attention scores between channels; (ii) the spatial attention refines the local representations by performing fine-grained interactions across spatial locations, which in turn helps the global information modeling in channel attention. Extensive experiments show our DaViT achieves state-of-the-art performance on four different tasks with efficient computations. Without extra data, DaViT-Tiny, DaViT-Small, and DaViT-Base achieve 82.8%, 84.2%, and 84.6% top-1 accuracy on ImageNet-1K with 28.3M, 49.7M, and 87.9M parameters, respectively. When we further scale up DaViT with 1.5B weakly supervised image and text pairs, DaViT-Gaint reaches 90.4% top-1 accuracy on ImageNet-1K. Code is available at https://github.com/dingmyu/davit.

DaViT: Dual Attention Vision Transformers

Published in

European Conference on Computer Vision(2022)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

References87 items

Unified Contrastive Learning in Image-Text-Label Space

Visual attention network

UniFormer: Unifying Convolution and Self-Attention for Visual Recognition

Patches Are All You Need?

A ConvNet for the 2020s

QuadTree Attention for Vision Transformers

ELSA: Enhanced Local Self-Attention for Vision Transformer

MViTv2: Improved Multiscale Vision Transformers for Classification and Detection

Florence: A New Foundation Model for Computer Vision

HRFormer: High-Resolution Transformer for Dense Prediction

UFO-ViT: High Performance Linear Vision Transformer without Softmax

Rethinking and Improving Relative Position Encoding for Vision Transformer

CycleMLP: A MLP-like Architecture for Dense Prediction

AS-MLP: An Axial Shifted MLP Architecture for Vision

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

Focal Self-attention for Local-Global Interactions in Vision Transformers

PVT v2: Improved baselines with Pyramid Vision Transformer

VOLO: Vision Outlooker for Visual Recognition

Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition

TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?

XCiT: Cross-Covariance Image Transformers

S2-MLP: Spatial-Shift MLP Architecture for Vision

Scaling Vision with Sparse Mixture of Experts

CoAtNet: Marrying Convolution and Attention for All Data Sizes

Scaling Vision Transformers

Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Glance-and-Gaze Vision Transformer

HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers

Dynamic Head: Unifying Object Detection Heads with Attentions

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

Pay Attention to MLPs

ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training

MLP-Mixer: An all-MLP Architecture for Vision

Visformer: The Vision-friendly Transformer

All Tokens Matter: Token Labeling for Training Better Vision Transformers

Co-Scale Conv-Attentional Image Transformers

LocalViT: Analyzing Locality in Vision Transformers

LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference

Going deeper with Image Transformers

Rethinking Spatial Dimensions of Vision Transformers

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

CvT: Introducing Convolutions to Vision Transformers

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

Learning Versatile Neural Architectures by Propagating Network Codes

Vision Transformers for Dense Prediction

Scaling Local Self-Attention for Parameter Efficient Visual Backbones

BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search

DeepViT: Towards Deeper Vision Transformer

Scalable Visual Transformers with Hierarchical Pooling

Transformer in Transformer

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Conditional Positional Encodings for Vision Transformers

LambdaNetworks: Modeling Long-Range Interactions Without Attention

High-Performance Large-Scale Image Recognition Without Normalization

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Bottleneck Transformers for Visual Recognition

Training data-efficient image transformers & distillation through attention

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Generative Pretraining From Pixels

Augment Your Batch: Improving Generalization Through Instance Repetition

ResNeSt: Split-Attention Networks

ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

Object-Contextual Representations for Semantic Segmentation

Deep High-Resolution Representation Learning for Visual Recognition

MultiGrain: a unified image embedding for classes and instances

Dual Attention Network for Scene Segmentation

Unified Perceptual Parsing for Scene Understanding

Super-convergence: very fast training of neural networks using large learning rates

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Decoupled Weight Decay Regularization

Squeeze-and-Excitation Networks

Focal Loss for Dense Object Detection

Scene Parsing through ADE20K Dataset

Attention is All you Need

Mask R-CNN

Feature Pyramid Networks for Object Detection

Aggregated Residual Transformations for Deep Neural Networks

Deep Residual Learning for Image Recognition

Very Deep Convolutional Networks for Large-Scale Image Recognition

Microsoft COCO: Common Objects in Context

ImageNet: A large-scale hierarchical image database

Acceleration of stochastic approximation by averaging

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Twins: Revisiting Spatial Attention Design in Vision Transformers

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Pytorch image models