MogaNet: Multi-order Gated Aggregation Network

Published in

International Conference on Learning Representa...(2022)

External Links:

Generate Graph

TL;DR

MogaNet encapsulates conceptually simple yet effective convolutions and gated aggregation into a compact module, where discriminative features are efficiently gathered and contextualized adaptively and exhibits great scalability, impressive efficiency of parameters, and competitive performance compared to state-of-the-art ViTs and ConvNets on ImageNet and various downstream vision benchmarks.

Abstract

By contextualizing the kernel as global as possible, Modern ConvNets have shown great potential in computer vision tasks. However, recent progress on multi-order game-theoretic interaction within deep neural networks (DNNs) reveals the representation bottleneck of modern ConvNets, where the expressive interactions have not been effectively encoded with the increased kernel size. To tackle this challenge, we propose a new family of modern ConvNets, dubbed MogaNet, for discriminative visual representation learning in pure ConvNet-based models with favorable complexity-performance trade-offs. MogaNet encapsulates conceptually simple yet effective convolutions and gated aggregation into a compact module, where discriminative features are efficiently gathered and contextualized adaptively. MogaNet exhibits great scalability, impressive efficiency of parameters, and competitive performance compared to state-of-the-art ViTs and ConvNets on ImageNet and various downstream vision benchmarks, including COCO object detection, ADE20K semantic segmentation, 2D&3D human pose estimation, and video prediction. Notably, MogaNet hits 80.0% and 87.8% accuracy with 5.2M and 181M parameters on ImageNet-1K, outperforming ParC-Net and ConvNeXt-L, while saving 59% FLOPs and 17M parameters, respectively. The source code is available at https://github.com/Westlake-AI/MogaNet.

Authors

Zicheng Liu

15 papers

Jiangbin Zheng

3 papers

Cheng Tan

5 papers

References158 items

Masked Modeling for Self-supervised Representation Learning on Vision and Beyond

Convolutional Networks with Oriented 1D Kernels

OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders

Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition

MogaNet: Multi-order Gated Aggregation Network

Published in

International Conference on Learning Representa...(2022)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Zicheng Liu

15 papers

Jiangbin Zheng

3 papers

Cheng Tan

5 papers

References158 items

Masked Modeling for Self-supervised Representation Learning on Vision and Beyond

Convolutional Networks with Oriented 1D Kernels

OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders

Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition

Zedong Wang

2 papers

Zhiyuan Chen

1 papers

OpenMixup: Open Mixup Toolbox and Benchmark for Visual Representation Learning

HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions

An Impartial Take to the CNN vs Transformer Robustness Contest

TinyViT: Fast Pretraining Distillation for Small Vision Transformers

More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity

DLME: Deep Local-flatness Manifold Embedding

EfficientFormer: Vision Transformers at MobileNet Speed

SimVP: Simpler yet Better Video Prediction

Architecture-Agnostic Masked Image Modeling - From ViT back to CNN

Fast Vision Transformers with HiLo Attention

Inception Transformer

Understanding The Robustness in Vision Transformers

DeiT III: Revenge of the ViT

Focal Modulation Networks

Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice

CF-ViT: A General Coarse-to-Fine Method for Vision Transformer

Transformer Quality in Linear Time

Visual attention network

How Do Vision Transformers Work?

Flowformer: Linearizing Transformers with Conservation Flows

When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism

UniFormer: Unifying Convolution and Self-Attention for Visual Recognition

A ConvNet for the 2020s

Augmenting Convolutional networks with attention-based aggregation

Reflash Dropout in Image Super-Resolution

RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality

A-ViT: Adaptive Tokens for Efficient Vision Transformer

Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup

MetaFormer is Actually What You Need for Vision

iBOT: Image BERT Pre-Training with Online Tokenizer

Discovering and Explaining the Representation Bottleneck of DNNs

Masked Autoencoders Are Scalable Vision Learners

GenURL: A General Framework for Unsupervised Representation Learning

MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

ResNet strikes back: An improved training procedure in timm

Do Vision Transformers See Like Convolutional Neural Networks?

Mobile-Former: Bridging MobileNet and Transformer

Rethinking and Improving Relative Position Encoding for Vision Transformer

CMT: Convolutional Neural Networks Meet Vision Transformers

Early Convolutions Help Transformers See Better

PVT v2: Improved baselines with Pyramid Vision Transformer

Video Swin Transformer

A Game-Theoretic Taxonomy of Visual Concepts in DNNs

BEiT: BERT Pre-Training of Image Transformers

Partial success in closing the gap between human and machine vision

CoAtNet: Marrying Convolution and Attention for All Data Sizes

Less is More: Pay Less Attention in Vision Transformers

Intriguing Properties of Vision Transformers

Are Convolutional Neural Networks or Transformers more like human vision?

MLP-Mixer: An all-MLP Architecture for Vision

Emerging Properties in Self-Supervised Vision Transformers

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

All Tokens Matter: Token Labeling for Training Better Vision Transformers

Multiscale Vision Transformers

LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference

EfficientNetV2: Smaller Models and Faster Training

ViViT: A Video Vision Transformer

CvT: Introducing Convolutions to Vision Transformers

AutoMix: Unveiling the Power of Mixup for Stronger Classifiers

ConViT: improving vision transformers with soft convolutional inductive biases

Transformer in Transformer

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up

High-Performance Large-Scale Image Recognition Without Normalization

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Characterizing signal propagation to close the performance gap in unnormalized ResNets

Training data-efficient image transformers & distillation through attention

Pre-Trained Image Processing Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Deformable DETR: Deformable Transformers for End-to-End Object Detection

Interpreting and Boosting Dropout from a Game-Theoretic View

Monocular Expressive Body Regression through Body-Driven Attention

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

Linformer: Self-Attention with Linear Complexity

Augment Your Batch: Improving Generalization Through Instance Repetition

Language Models are Few-Shot Learners

End-to-End Object Detection with Transformers

ResNeSt: Split-Attention Networks

Designing Network Design Spaces

GLU Variants Improve Transformer

Orthogonal Convolutional Neural Networks

The Origins and Prevalence of Texture Bias in Convolutional Neural Networks

Momentum Contrast for Unsupervised Visual Representation Learning

Randaugment: Practical automated data augmentation with a reduced search space

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

Cascade R-CNN: High Quality Object Detection and Instance Segmentation

MMDetection: Open MMLab Detection Toolbox and Benchmark

Learning Deep Transformer Models for Machine Translation

AutoAugment: Learning Augmentation Strategies From Data

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Searching for MobileNetV3

GCNet: Non-Local Networks Meet Squeeze-Excitation Networks and Beyond

100

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

101

Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation

102

Panoptic Feature Pyramid Networks

103

A Style-Based Generator Architecture for Generative Adversarial Networks

104

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

105

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

106

Unified Perceptual Parsing for Scene Understanding

107

CBAM: Convolutional Block Attention Module

108

Simple Baselines for Human Pose Estimation and Tracking

109

Evaluation of Dense 3D Reconstruction from 2D Face Images in the Wild

110

Image Transformer

111

MobileNetV2: Inverted Residuals and Linear Bottlenecks

112

Non-local Neural Networks

113

Decoupled Weight Decay Regularization

114

mixup: Beyond Empirical Risk Minimization

115

Squeeze-and-Excitation Networks

116

Random Erasing Data Augmentation

117

Focal Loss for Dense Object Detection

118

Attention is All you Need

119

Train longer, generalize better: closing the generalization gap in large batch training of neural networks

120

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

122

Deformable Convolutional Networks

123

Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning

124

Feature Pyramid Networks for Object Detection

125

Understanding the Effective Receptive Field in Deep Convolutional Neural Networks

126

Aggregated Residual Transformations for Deep Neural Networks

127

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

128

Semantic Understanding of Scenes Through the ADE20K Dataset

129

SGDR: Stochastic Gradient Descent with Warm Restarts

130

Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units

131

Wide Residual Networks

132

Deep Networks with Stochastic Depth

133

Deep Residual Learning for Image Recognition

134

Rethinking the Inception Architecture for Computer Vision

135

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

136

Unsupervised Learning of Video Representations using LSTMs

137

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

138

Going deeper with convolutions

139

Very Deep Convolutional Networks for Large-Scale Image Recognition

140

Neural Machine Translation by Jointly Learning to Align and Translate

141

Microsoft COCO: Common Objects in Context

142

ImageNet classification with deep convolutional neural networks

143

ImageNet: A large-scale hierarchical image database

144

Cryptosporidium Parvum Genome Project

145

Acceleration of stochastic approximation by averaging

146

A feature-integration theory of attention

147

Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs

148

EdgeFormer: Improving Light-weight ConvNets by Learning from Vision Transformers

149

Discovering the Representation Bottleneck of Graph Neural Networks from Multi-order Interactions

150

Decoupled Mixup for Data-efficient Learning

151

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

152

Demystifying Local Vision Transformer: Sparse Connectivity, Weight Sharing, and Dynamic Weight

153

Rethinking ”batch” in

154

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

155

A method for stochastic optimization

156

Gradient-based learning applied to document recognition

157

MMHuman3D Contributors

158

Patches are all you need? ArXiv ,

Field of Study

Computer Science

Venue Information

Name

International Conference on Learning Representations

Type

conference

URL

https://iclr.cc/

Alternate Names

Int Conf Learn Represent
ICLR