VanillaNet: the Power of Minimalism in Deep Learning (2023-05-22T00:00:00.000000Z)

TL;DR

This study introduces VanillaNet, a neural network architecture that embraces elegance in design and delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning.

Abstract

At the heart of foundation models is the philosophy of"more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https://github.com/huawei-noah/VanillaNet and https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet.

Authors

Yunhe Wang

12 papers

Hanting Chen

5 papers

Jianyuan Guo

5 papers

TL;DR

Abstract

Authors

References64 items

VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale

BNET: Batch Normalization With Enhanced Linear Transformation

ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders

DeepNet: Scaling Transformers to 1,000 Layers

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

A ConvNet for the 2020s

ResNet strikes back: An improved training procedure in timm

AS-MLP: An Axial Shifted MLP Architecture for Vision

CMT: Convolutional Neural Networks Meet Vision Transformers

BEiT: BERT Pre-Training of Image Transformers

Scaling Vision Transformers

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Universal Adder Neural Networks

Going deeper with Image Transformers

Distilling Object Detectors via Decoupled Features

Diverse Branch Block: Building a Convolution as an Inception-like Unit

RepVGG: Making VGG-style ConvNets Great Again

Training data-efficient image transformers & distillation through attention

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Are we done with ImageNet?

Language Models are Few-Shot Learners

Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection

Randaugment: Practical automated data augmentation with a reduced search space

ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks

Quantization Networks

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Searching for MobileNetV3

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks

mixup: Beyond Empirical Risk Minimization

Squeeze-and-Excitation Networks

Random Erasing Data Augmentation

Focal Loss for Dense Object Detection

The iNaturalist Species Classification and Detection Dataset

Attention is All you Need

Mask R-CNN

Aggregated Residual Transformations for Deep Neural Networks

Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates

SGDR: Stochastic Gradient Descent with Warm Restarts

Deep vs. shallow networks : An approximation theory perspective

Gaussian Error Linear Units (GELUs)

Identity Mappings in Deep Residual Networks

Benefits of Depth in Neural Networks

The Power of Depth for Feedforward Neural Networks

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Fully convolutional networks for semantic segmentation

Very Deep Convolutional Networks for Large-Scale Image Recognition

Microsoft COCO: Common Objects in Context

New types of deep neural network learning for speech recognition and related applications: an overview

ImageNet classification with deep convolutional neural networks

Towards fully autonomous driving: Systems and algorithms

ImageNet: A large-scale hierarchical image database

Acceleration of stochastic approximation by averaging

Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts

Electra: Pre-training text encoders as discriminators rather than generators

Searching for Activation Functions

Face recognition: a convolutional neural-network approach

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names