Visual attention network (2022-02-20T00:00:00.000000Z)

TL;DR

This paper proposes a novel linear attention named large kernel attention (LKA) to enable self-adaptive and long-range correlations in self-attention while avoiding its shortcomings, and presents a neural network based on LKA, namely Visual Attention Network (VAN).

Abstract

While originally designed for natural language processing tasks, the self-attention mechanism has recently taken various computer vision areas by storm. However, the 2D nature of images brings three challenges for applying self-attention in computer vision: (1) treating images as 1D sequences neglects their 2D structures; (2) the quadratic complexity is too expensive for high-resolution images; (3) it only captures spatial adaptability but ignores channel adaptability. In this paper, we propose a novel linear attention named large kernel attention (LKA) to enable self-adaptive and long-range correlations in self-attention while avoiding its shortcomings. Furthermore, we present a neural network based on LKA, namely Visual Attention Network (VAN). While extremely simple, VAN achieves comparable results with similar size convolutional neural networks (CNNs) and vision transformers (ViTs) in various tasks, including image classification, object detection, semantic segmentation, panoptic segmentation, pose estimation, etc. For example, VAN-B6 achieves 87.8% accuracy on ImageNet benchmark, and sets new state-of-the-art performance (58.2 PQ) for panoptic segmentation. Besides, VAN-B2 surpasses Swin-T 4 mIoU (50.1 vs. 46.1) for semantic segmentation on ADE20K benchmark, 2.6 AP (48.8 vs. 46.2) for object detection on COCO dataset. It provides a novel method and a simple yet strong baseline for the community. The code is available at https://github.com/Visual-Attention-Network.

Authors

Meng-Hao Guo

5 papers

Zheng-Ning Liu

4 papers

Chengrou Lu

1 papers

TL;DR

Abstract

Authors

References133 items

MixFormer: Mixing Features across Windows and Dimensions

Focal Modulation Networks

Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs

DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR

A ConvNet for the 2020s

MPViT: Multi-Path Vision Transformer for Dense Prediction

Masked-attention Mask Transformer for Universal Image Segmentation

MetaFormer is Actually What You Need for Vision

Attention mechanisms in computer vision: A survey

Masked Autoencoders Are Scalable Vision Learners

Are we ready for a new paradigm shift? A survey on visual deep MLP

Transformers in computational visual media: A survey

Is Attention Better Than Matrix Decomposition?

FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting

Query2Label: A Simple Transformer Way to Multi-Label Classification

CMT: Convolutional Neural Networks Meet Vision Transformers

Focal Self-attention for Local-Global Interactions in Vision Transformers

PVT v2: Improved baselines with Pyramid Vision Transformer

P2T: Pyramid Pooling Transformer for Scene Understanding

XCiT: Cross-Covariance Image Transformers

BEiT: BERT Pre-Training of Image Transformers

CoAtNet: Marrying Convolution and Attention for All Data Sizes

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

Can attention enable MLPs to catch up with CNNs?

Pay Attention to MLPs

ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training

Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks

MLP-Mixer: An all-MLP Architecture for Vision

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

Decoupled Spatial-Temporal Transformer for Video Inpainting

Co-Scale Conv-Attentional Image Transformers

EfficientNetV2: Smaller Models and Faster Training

Going deeper with Image Transformers

CvT: Introducing Convolutions to Vision Transformers

Revisiting ResNets: Improved Training and Scaling Strategies

Transformer in Transformer

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

LambdaNetworks: Modeling Long-Range Interactions Without Attention

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Bottleneck Transformers for Visual Recognition

EDN: Salient Object Detection via Extremely-Downsampled Network

Training data-efficient image transformers & distillation through attention

FcaNet: Frequency Channel Attention Networks

PCT: Point cloud transformer

Sparse R-CNN: End-to-End Object Detection with Learnable Proposals

Jittor: a novel deep learning framework with meta-operators and unified graph execution

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection

D2Det: Towards High Quality Object Detection and Instance Segmentation

Language Models are Few-Shot Learners

End-to-End Object Detection with Transformers

Designing Network Design Spaces

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection

PyTorch: An Imperative Style, High-Performance Deep Learning Library

ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

Object-Contextual Representations for Semantic Segmentation

Deep High-Resolution Representation Learning for Visual Recognition

Cascade R-CNN: High Quality Object Detection and Instance Segmentation

MMDetection: Open MMLab Detection Toolbox and Benchmark

Stand-Alone Self-Attention in Vision Models

When Does Label Smoothing Help?

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Attention Augmented Convolutional Networks

Res2Net: A New Multi-Scale Backbone Architecture

Panoptic Feature Pyramid Networks

CCNet: Criss-Cross Attention for Semantic Segmentation

Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks

Dual Attention Network for Scene Segmentation

OCNet: Object Context Network for Scene Parsing

Unified Perceptual Parsing for Scene Understanding

CBAM: Convolutional Block Attention Module

BAM: Bottleneck Attention Module

Attentional ShapeContextNet for Point Cloud Recognition

Self-Attention Generative Adversarial Networks