ResNeSt: Split-Attention Networks

Published in

(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

This work designs a new variant of the ResNet model, named ResNeSt, which outperforms EfficientNet in terms of the accuracy/latency trade-off and applies channel-wise attention across different network branches to leverage the complementary strengths of both feature-map attention and multi-path representation.

Abstract

The ability to learn richer network representations generally boosts the performance of deep learning models. To improve representation-learning in convolutional neural networks, we present a multi-branch architecture, which applies channel-wise attention across different network branches to leverage the complementary strengths of both feature-map attention and multi-path representation. Our proposed Split-Attention module provides a simple and modular computation block that can serve as a drop-in replacement for the popular residual block, while producing more diverse representations via cross-feature interactions. Adding a Split-Attention module into the architecture design space of RegNet-Y and FBNetV2 directly improves the performance of the resulting network. Replacing residual blocks with our Split-Attention module, we further design a new variant of the ResNet model, named ResNeSt, which outperforms EfficientNet in terms of the accuracy/latency trade-off.

Authors

Jonas W. Mueller

6 papers

Haibin Lin

3 papers

Alex Smola

9 papers

References82 items

FP-NAS: Fast Probabilistic Neural Architecture Search

FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions

Designing Network Design Spaces

Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network

Adaptive Context Network for Scene Parsing

ResNeSt: Split-Attention Networks

Published in

(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Jonas W. Mueller

6 papers

Haibin Lin

3 papers

Alex Smola

9 papers

References82 items

FP-NAS: Fast Probabilistic Neural Architecture Search

FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions

Designing Network Design Spaces

Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network

Adaptive Context Network for Scene Parsing

Zhongyue Zhang

3 papers

R. Manmatha

3 papers

Zhi-Li Zhang

2 papers

Chongruo Wu

2 papers

Object-Contextual Representations for Semantic Segmentation

Once for All: Train One Network and Specialize it for Efficient Deployment

GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing

Cascade R-CNN: High Quality Object Detection and Instance Segmentation

MMDetection: Open MMLab Detection Toolbox and Benchmark

Co-Occurrent Features in Semantic Segmentation

AutoAugment: Learning Augmentation Strategies From Data

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Searching for MobileNetV3

Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources

Selective Kernel Networks

Hybrid Task Cascade for Instance Segmentation

FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search

Bag of Tricks for Image Classification with Convolutional Neural Networks

Improving Semantic Segmentation via Video Propagation and Label Relaxation

Deformable ConvNets V2: More Deformable, Better Results

DropBlock: A regularization method for convolutional networks

Dual Attention Network for Scene Segmentation

PSANet: Point-wise Spatial Attention Network for Scene Parsing

OCNet: Object Context Network for Scene Parsing

Reversible Inactivation of Different Millimeter-Scale Regions of Primate IT Results in Different Patterns of Core Object Recognition Deficits

MnasNet: Platform-Aware Neural Architecture Search for Mobile

Unified Perceptual Parsing for Scene Understanding

DARTS: Differentiable Architecture Search

SNIPER: Efficient Multi-Scale Training

Simple Baselines for Human Pose Estimation and Tracking

Adaptive Affinity Fields for Semantic Segmentation

Context Encoding for Semantic Segmentation

Efficient Neural Architecture Search via Parameter Sharing

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Regularized Evolution for Image Classifier Architecture Search

Cascade R-CNN: Delving Into High Quality Object Detection

Non-local Neural Networks

mixup: Beyond Empirical Risk Minimization

Squeeze-and-Excitation Networks

Learning Transferable Architectures for Scalable Image Recognition

Scene Parsing through ADE20K Dataset

Rethinking Atrous Convolution for Semantic Image Segmentation

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Deformable Convolutional Networks

Feature Pyramid Networks for Object Detection

Pyramid Scene Parsing Network

RMPE: Regional Multi-person Pose Estimation

PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Aggregated Residual Transformations for Deep Neural Networks

Densely Connected Convolutional Networks

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

The Cityscapes Dataset for Semantic Urban Scene Understanding

Stacked Hourglass Networks for Human Pose Estimation

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks

Deep Residual Learning for Image Recognition

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

Rethinking the Inception Architecture for Computer Vision

Multi-Scale Context Aggregation by Dilated Convolutions

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Learning Deep Features for Scene Recognition using Places Database

Fully convolutional networks for semantic segmentation

Going deeper with convolutions

Very Deep Convolutional Networks for Large-Scale Image Recognition

Understanding Vision: Theory, Models, and Data

Microsoft COCO: Common Objects in Context

Network In Network

ImageNet classification with deep convolutional neural networks

Rectified Linear Units Improve Restricted Boltzmann Machines

ImageNet: A large-scale hierarchical image database

Proposal Scaling Distributed Machine Learning with System and Algorithm Co-design

Dropout: a simple way to prevent neural networks from overfitting

Gradient-based learning applied to document recognition

Receptive fields, binocular interaction and functional architecture in the cat's visual cortex

Tensorflow Efficientnet

Field of Study

Computer Science

Journal Information

Name

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Page

456-465

TL;DR

Abstract

Authors

References82 items

FP-NAS: Fast Probabilistic Neural Architecture Search

FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions

Designing Network Design Spaces

Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network

Adaptive Context Network for Scene Parsing

TL;DR

Abstract

Authors

References82 items

FP-NAS: Fast Probabilistic Neural Architecture Search

FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions

Designing Network Design Spaces

Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network

Adaptive Context Network for Scene Parsing

Object-Contextual Representations for Semantic Segmentation

Once for All: Train One Network and Specialize it for Efficient Deployment

GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing

Cascade R-CNN: High Quality Object Detection and Instance Segmentation

MMDetection: Open MMLab Detection Toolbox and Benchmark

Co-Occurrent Features in Semantic Segmentation

AutoAugment: Learning Augmentation Strategies From Data

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Searching for MobileNetV3

Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources

Selective Kernel Networks

Hybrid Task Cascade for Instance Segmentation

FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search

Bag of Tricks for Image Classification with Convolutional Neural Networks

Improving Semantic Segmentation via Video Propagation and Label Relaxation

Deformable ConvNets V2: More Deformable, Better Results

DropBlock: A regularization method for convolutional networks

Dual Attention Network for Scene Segmentation

PSANet: Point-wise Spatial Attention Network for Scene Parsing

OCNet: Object Context Network for Scene Parsing

Reversible Inactivation of Different Millimeter-Scale Regions of Primate IT Results in Different Patterns of Core Object Recognition Deficits

MnasNet: Platform-Aware Neural Architecture Search for Mobile

Unified Perceptual Parsing for Scene Understanding

DARTS: Differentiable Architecture Search

SNIPER: Efficient Multi-Scale Training

Simple Baselines for Human Pose Estimation and Tracking

Adaptive Affinity Fields for Semantic Segmentation

Context Encoding for Semantic Segmentation

Efficient Neural Architecture Search via Parameter Sharing

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Regularized Evolution for Image Classifier Architecture Search

Cascade R-CNN: Delving Into High Quality Object Detection

Non-local Neural Networks

mixup: Beyond Empirical Risk Minimization

Squeeze-and-Excitation Networks

Learning Transferable Architectures for Scalable Image Recognition

Scene Parsing through ADE20K Dataset

Rethinking Atrous Convolution for Semantic Image Segmentation

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Mask R-CNN

Deformable Convolutional Networks

Feature Pyramid Networks for Object Detection

Pyramid Scene Parsing Network

RMPE: Regional Multi-person Pose Estimation

PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Aggregated Residual Transformations for Deep Neural Networks

Densely Connected Convolutional Networks

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

The Cityscapes Dataset for Semantic Urban Scene Understanding

Stacked Hourglass Networks for Human Pose Estimation

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks

Deep Residual Learning for Image Recognition

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

Rethinking the Inception Architecture for Computer Vision

Multi-Scale Context Aggregation by Dilated Convolutions

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Highway Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Learning Deep Features for Scene Recognition using Places Database