EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Published in

International Conference on Machine Learning(2019)

External Links:

Generate Graph

TL;DR

A new scaling method is proposed that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient and is demonstrated the effectiveness of this method on scaling up MobileNets and ResNet.

Abstract

Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet. To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.4% top-1 / 97.1% top-5 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters. Source code is at this https URL.

Authors

Quoc V. Le

42 papers

Mingxing Tan

7 papers

References54 items

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Domain Adaptive Transfer Learning with Specialist Models

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

Neural Architecture Optimization

MnasNet: Platform-Aware Neural Architecture Search for Mobile

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Published in

International Conference on Machine Learning(2019)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Quoc V. Le

42 papers

Mingxing Tan

7 papers

References54 items

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Domain Adaptive Transfer Learning with Specialist Models

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

Neural Architecture Optimization

MnasNet: Platform-Aware Neural Architecture Search for Mobile

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

ResNet with one-neuron hidden layers is a Universal Approximator

AutoAugment: Learning Augmentation Policies from Data

Do Better ImageNet Models Transfer Better?

Exploring the Limits of Weakly Supervised Pretraining

NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications

SqueezeNext: Hardware-Aware Neural Network Design

AMC: AutoML for Model Compression and Acceleration on Mobile Devices

Regularized Evolution for Image Classifier Architecture Search

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Progressive Neural Architecture Search

Swish: a Self-Gated Activation Function

The Expressive Power of Neural Networks: A View from the Width

Squeeze-and-Excitation Networks

Learning Transferable Architectures for Scalable Image Recognition

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

On the Expressive Power of Overlapping Architectures of Deep Learning

Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning

Feature Pyramid Networks for Object Detection

PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Aggregated Residual Transformations for Deep Neural Networks

Neural Architecture Search with Reinforcement Learning

Xception: Deep Learning with Depthwise Separable Convolutions

Densely Connected Convolutional Networks

Gaussian Error Linear Units (GELUs)

On the Expressive Power of Deep Neural Networks

Wide Residual Networks

Deep Networks with Stochastic Depth

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Learning Deep Features for Discriminative Localization

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Going deeper with convolutions

Food-101 - Mining Discriminative Components with Random Forests

ImageNet Large Scale Visual Recognition Challenge

Birdsnap: Large-Scale Fine-Grained Visual Categorization of Birds

Fine-Grained Visual Classification of Aircraft

ImageNet classification with deep convolutional neural networks

Automated Flower Classification over a Large Number of Classes

Care of aged doctors

Searching for Activation Functions

Dropout: a simple way to prevent neural networks from overfitting

Collecting a Large-scale Dataset of Fine-grained Cars

Learning Multiple Layers of Features from Tiny Images

Field of Study

Computer ScienceMathematics

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

International Conference on Machine Learning

Type

conference

URL

https://icml.cc/

Alternate Names

ICML
Int Conf Mach Learn

TL;DR

Abstract

Authors

References54 items

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Domain Adaptive Transfer Learning with Specialist Models

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

Neural Architecture Optimization

MnasNet: Platform-Aware Neural Architecture Search for Mobile

TL;DR

Abstract

Authors

References54 items

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Domain Adaptive Transfer Learning with Specialist Models

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

Neural Architecture Optimization

MnasNet: Platform-Aware Neural Architecture Search for Mobile

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

ResNet with one-neuron hidden layers is a Universal Approximator

AutoAugment: Learning Augmentation Policies from Data

Do Better ImageNet Models Transfer Better?

Exploring the Limits of Weakly Supervised Pretraining

NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications

SqueezeNext: Hardware-Aware Neural Network Design

AMC: AutoML for Model Compression and Acceleration on Mobile Devices

Regularized Evolution for Image Classifier Architecture Search

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Progressive Neural Architecture Search

Swish: a Self-Gated Activation Function

The Expressive Power of Neural Networks: A View from the Width

Squeeze-and-Excitation Networks

Learning Transferable Architectures for Scalable Image Recognition

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Mask R-CNN

On the Expressive Power of Overlapping Architectures of Deep Learning

Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning

Feature Pyramid Networks for Object Detection

PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Aggregated Residual Transformations for Deep Neural Networks

Neural Architecture Search with Reinforcement Learning

Xception: Deep Learning with Depthwise Separable Convolutions

Densely Connected Convolutional Networks

Gaussian Error Linear Units (GELUs)

On the Expressive Power of Deep Neural Networks

Wide Residual Networks

Deep Networks with Stochastic Depth

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Learning Deep Features for Discriminative Localization

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Going deeper with convolutions

Food-101 - Mining Discriminative Components with Random Forests

ImageNet Large Scale Visual Recognition Challenge

Birdsnap: Large-Scale Fine-Grained Visual Categorization of Birds

Fine-Grained Visual Classification of Aircraft

ImageNet classification with deep convolutional neural networks

Automated Flower Classification over a Large Number of Classes

Care of aged doctors

Searching for Activation Functions

Dropout: a simple way to prevent neural networks from overfitting

Collecting a Large-scale Dataset of Fine-grained Cars

Learning Multiple Layers of Features from Tiny Images

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names