Demystify Transformers & Convolutions in Modern Image Deep Networks (2022-11-10T00:00:00.000000Z)

TL;DR

This paper aims to identify the real gains of popular convolution and attention operators through a detailed study, and finds that the key difference among these feature transformation modules, such as attention or convolution, lies in their spatial feature aggregation approach, known as the “spatial token mixer” (STM).

Abstract

Vision transformers have gained popularity recently, leading to the development of new vision backbones with improved features and consistent performance gains. However, these advancements are not solely attributable to novel feature transformation designs; certain benefits also arise from advanced network-level and block-level architectures. This paper aims to identify the real gains of popular convolution and attention operators through a detailed study. We find that the key difference among these feature transformation modules, such as attention or convolution, lies in their spatial feature aggregation approach, known as the “spatial token mixer” (STM). To facilitate an impartial comparison, we introduce a unified architecture to neutralize the impact of divergent network-level and block-level designs. Subsequently, various STMs are integrated into this unified framework for comprehensive comparative analysis. Our experiments on various tasks and an analysis of inductive bias show a significant performance boost due to advanced network-level and block-level designs, but performance differences persist among different STMs. Our detailed analysis also reveals various findings about different STMs, including effective receptive fields, invariance, and adversarial robustness tests.

Authors

Wenhai Wang

14 papers

Xizhou Zhu

7 papers

Y. Qiao

16 papers

TL;DR

Abstract

Authors

References60 items

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions

MetaFormer Baselines for Vision

HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions

More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity

Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs

A ConvNet for the 2020s

Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention

MetaFormer is Actually What You Need for Vision

Swin Transformer V2: Scaling Up Capacity and Resolution

ResNet strikes back: An improved training procedure in timm

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

PVT v2: Improved baselines with Pyramid Vision Transformer

CoAtNet: Marrying Convolution and Attention for All Data Sizes

On the Connection between Local Attention and Dynamic Depth-wise Convolution

Going deeper with Image Transformers

Scaling Local Self-Attention for Parameter Efficient Visual Backbones

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Training data-efficient image transformers & distillation through attention

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Linformer: Self-Attention with Linear Complexity

Randaugment: Practical automated data augmentation with a reduced search space

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Making Convolutional Networks Shift-Invariant Again

Deformable ConvNets V2: More Deformable, Better Results

Non-local Neural Networks

Decoupled Weight Decay Regularization

Dynamic Routing Between Capsules

mixup: Beyond Empirical Risk Minimization

Squeeze-and-Excitation Networks

Random Erasing Data Augmentation

Attention is All you Need

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Mask R-CNN

Deformable Convolutional Networks

Understanding the Effective Receptive Field in Deep Convolutional Neural Networks

Aggregated Residual Transformations for Deep Neural Networks

Densely Connected Convolutional Networks

Deep Networks with Stochastic Depth

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Explaining and Harnessing Adversarial Examples

Going deeper with convolutions

Very Deep Convolutional Networks for Large-Scale Image Recognition

Microsoft COCO: Common Objects in Context

ImageNet classification with deep convolutional neural networks

ImageNet: A large-scale hierarchical image database

Pattern

Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

SPIKING NEURAL NETWORKS

“Pytorch image models,”

Deep Learning

are with Tsinghua University, Beijing 100084, China

is with Shanghai Artiﬁcial Intelligence Laboratory, Shanghai 201210, China, and also with The Chinese University of Hong Kong, Shatin, Hong Kong

Exploring large-scale vision fundamental models

China, and also with the Huazhong University of Science and Technology, Wuhan 430074, China

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names