Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets (2020-10-28T00:00:00.000000Z)

TL;DR

A tiny formula for downsizing neural architectures through a series of smaller models derived from the EfficientNet-B0 with the FLOPs constraint is summarized, observing that resolution and depth are more important than width for tiny networks.

Abstract

To obtain excellent deep neural architectures, a series of techniques are carefully designed in EfficientNets. The giant formula for simultaneously enlarging the resolution, depth and width provides us a Rubik's cube for neural networks. So that we can find networks with high efficiency and excellent performance by twisting the three dimensions. This paper aims to explore the twisting rules for obtaining deep neural networks with minimum model sizes and computational costs. Different from the network enlarging, we observe that resolution and depth are more important than width for tiny networks. Therefore, the original method, i.e., the compound scaling in EfficientNet is no longer suitable. To this end, we summarize a tiny formula for downsizing neural architectures through a series of smaller models derived from the EfficientNet-B0 with the FLOPs constraint. Experimental results on the ImageNet benchmark illustrate that our TinyNet performs much better than the smaller version of EfficientNets using the inversed giant formula. For instance, our TinyNet-E achieves a 59.9% Top-1 accuracy with only 24M FLOPs, which is about 1.9% higher than that of the previous best MobileNetV3 with similar computational cost. Code will be available at this https URL, and this https URL.

Authors

Kai Han

7 papers

Yunhe Wang

12 papers

Chunjing Xu

7 papers

TL;DR

Abstract

Authors

References61 items

Training Binary Neural Networks through Learning with Noisy Supervision

GAN Slimming: All-in-One GAN Compression by A Unified Optimization Framework

Label Enhancement for Label Distribution Learning via Prior Knowledge

A Semi-Supervised Assessor of Neural Architectures

Reborn Filters: Pruning Convolutional Neural Networks with Limited Data

Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection

GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet

HRank: Filter Pruning Using High-Rank Feature Map

PyTorch: An Imperative Style, High-Performance Deep Learning Library

GhostNet: More Features From Cheap Operations

Self-Training With Noisy Student Improves ImageNet Classification

Randaugment: Practical automated data augmentation with a reduced search space

Efficient Residual Dense Block Search for Image Super-Resolution

Positive-Unlabeled Compression on the Cloud

CARS: Continuous Evolution for Efficient Neural Architecture Search

Co-Evolutionary Compression for Unpaired Image Translation

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

LegoNet: Efficient Convolutional Neural Networks with Lego Filters

Searching for MobileNetV3

Towards Optimal Structured CNN Pruning via Generative Adversarial Learning

FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search

Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation

Attribute-Aware Attention Model for Fine-grained Representation Learning

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm

MnasNet: Platform-Aware Neural Architecture Search for Mobile

ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design

DARTS: Differentiable Architecture Search

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Learning from Multiple Teacher Networks

On Compressing Deep Models by Low Rank and Sparse Decomposition

ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Neural Architecture Search with Reinforcement Learning

Pruning Filters for Efficient ConvNets

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Wide Residual Networks

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Learning Deep Features for Discriminative Localization

Deep Residual Learning for Image Recognition

SSD: Single Shot MultiBox Detector

Accurate Image Super-Resolution Using Very Deep Convolutional Networks

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Distilling the Knowledge in a Neural Network

FitNets: Hints for Thin Deep Nets

An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: Solving Problems With Box Constraints

Speeding up Convolutional Neural Networks with Low Rank Expansions

Microsoft COCO: Common Objects in Context

Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

ImageNet classification with deep convolutional neural networks

ImageNet: A large-scale hierarchical image database

Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)

Binarized Neural Networks

Gradient-based learning applied to document recognition

Tradeoff decision in multiple criteria decision making

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names