MogaNet: Multi-order Gated Aggregation Network (2022-11-07T00:00:00.000000Z)

TL;DR

MogaNet encapsulates conceptually simple yet effective convolutions and gated aggregation into a compact module, where discriminative features are efficiently gathered and contextualized adaptively and exhibits great scalability, impressive efficiency of parameters, and competitive performance compared to state-of-the-art ViTs and ConvNets on ImageNet and various downstream vision benchmarks.

Abstract

By contextualizing the kernel as global as possible, Modern ConvNets have shown great potential in computer vision tasks. However, recent progress on multi-order game-theoretic interaction within deep neural networks (DNNs) reveals the representation bottleneck of modern ConvNets, where the expressive interactions have not been effectively encoded with the increased kernel size. To tackle this challenge, we propose a new family of modern ConvNets, dubbed MogaNet, for discriminative visual representation learning in pure ConvNet-based models with favorable complexity-performance trade-offs. MogaNet encapsulates conceptually simple yet effective convolutions and gated aggregation into a compact module, where discriminative features are efficiently gathered and contextualized adaptively. MogaNet exhibits great scalability, impressive efficiency of parameters, and competitive performance compared to state-of-the-art ViTs and ConvNets on ImageNet and various downstream vision benchmarks, including COCO object detection, ADE20K semantic segmentation, 2D&3D human pose estimation, and video prediction. Notably, MogaNet hits 80.0% and 87.8% accuracy with 5.2M and 181M parameters on ImageNet-1K, outperforming ParC-Net and ConvNeXt-L, while saving 59% FLOPs and 17M parameters, respectively. The source code is available at https://github.com/Westlake-AI/MogaNet.

Authors

Zicheng Liu

15 papers

Jiangbin Zheng

3 papers

Cheng Tan

5 papers

TL;DR

Abstract

Authors

References158 items

Masked Modeling for Self-supervised Representation Learning on Vision and Beyond

Convolutional Networks with Oriented 1D Kernels

OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders

Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition

OpenMixup: Open Mixup Toolbox and Benchmark for Visual Representation Learning

HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions

An Impartial Take to the CNN vs Transformer Robustness Contest

TinyViT: Fast Pretraining Distillation for Small Vision Transformers

More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity

DLME: Deep Local-flatness Manifold Embedding

EfficientFormer: Vision Transformers at MobileNet Speed

SimVP: Simpler yet Better Video Prediction

Architecture-Agnostic Masked Image Modeling - From ViT back to CNN

Fast Vision Transformers with HiLo Attention

Inception Transformer

Understanding The Robustness in Vision Transformers

DeiT III: Revenge of the ViT

Focal Modulation Networks

Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice

CF-ViT: A General Coarse-to-Fine Method for Vision Transformer

Transformer Quality in Linear Time

Visual attention network

How Do Vision Transformers Work?

Flowformer: Linearizing Transformers with Conservation Flows

When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism

UniFormer: Unifying Convolution and Self-Attention for Visual Recognition

A ConvNet for the 2020s

Augmenting Convolutional networks with attention-based aggregation

Reflash Dropout in Image Super-Resolution

RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality

A-ViT: Adaptive Tokens for Efficient Vision Transformer

Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup

MetaFormer is Actually What You Need for Vision

iBOT: Image BERT Pre-Training with Online Tokenizer

Discovering and Explaining the Representation Bottleneck of DNNs

Masked Autoencoders Are Scalable Vision Learners

GenURL: A General Framework for Unsupervised Representation Learning

MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

ResNet strikes back: An improved training procedure in timm

Do Vision Transformers See Like Convolutional Neural Networks?

Mobile-Former: Bridging MobileNet and Transformer

Rethinking and Improving Relative Position Encoding for Vision Transformer

CMT: Convolutional Neural Networks Meet Vision Transformers

Early Convolutions Help Transformers See Better

PVT v2: Improved baselines with Pyramid Vision Transformer

Video Swin Transformer

A Game-Theoretic Taxonomy of Visual Concepts in DNNs

BEiT: BERT Pre-Training of Image Transformers

Partial success in closing the gap between human and machine vision

CoAtNet: Marrying Convolution and Attention for All Data Sizes

Less is More: Pay Less Attention in Vision Transformers

Intriguing Properties of Vision Transformers

Are Convolutional Neural Networks or Transformers more like human vision?

MLP-Mixer: An all-MLP Architecture for Vision

Emerging Properties in Self-Supervised Vision Transformers

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

All Tokens Matter: Token Labeling for Training Better Vision Transformers

Multiscale Vision Transformers

LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference

EfficientNetV2: Smaller Models and Faster Training

ViViT: A Video Vision Transformer

CvT: Introducing Convolutions to Vision Transformers

AutoMix: Unveiling the Power of Mixup for Stronger Classifiers

ConViT: improving vision transformers with soft convolutional inductive biases

Transformer in Transformer

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up

High-Performance Large-Scale Image Recognition Without Normalization

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Characterizing signal propagation to close the performance gap in unnormalized ResNets

Training data-efficient image transformers & distillation through attention

Pre-Trained Image Processing Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Interpreting and Boosting Dropout from a Game-Theoretic View