Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2021-06-16T00:00:00.000000Z)

TL;DR

This is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support and the seminal work there.

Abstract

Deep learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval, and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, and resources required to train, among others, have all increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. It is our hope that this survey would provide readers with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.

Authors

Gaurav Menghani

1 papers

TL;DR

Abstract

Authors

References180 items

Scaling Autoregressive Models for Content-Rich Text-to-Image Generation

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Hierarchical Text-Conditional Image Generation with CLIP Latents

PaLM: Scaling Language Modeling with Pathways

The Efficiency Misnomer

MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

The Bottleneck

A comprehensive survey on optimizing deep learning models by metaheuristics

Distilling Large Language Models into Tiny and Effective Students using pQRNN

Amazon SageMaker Automatic Model Tuning: Scalable Black-box Optimization

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Characterising Bias in Compressed Models

A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions

Efficient Transformers: A Survey

Towards Accurate Post-training Network Quantization via Bit-Split and Stitching

Neural Structured Learning: Training Neural Networks with Structured Signals

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Big Self-Supervised Models are Strong Semi-Supervised Learners

Language Models are Few-Shot Learners

Exploring Bayesian Optimization

Training with Quantization Noise for Extreme Model Compression

ProFormer: Towards On-Device LSH Projection Based Transformers

MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

Hyper-Parameter Optimization: A Review of Algorithms and Applications

Multi-modal Self-Supervision from Generalized Data Transformations

A Simple Framework for Contrastive Learning of Visual Representations

TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Rigging the Lottery: Making All Tickets Winners

Fast Sparse ConvNets

Mastering Atari, Go, chess and shogi by planning with a learned model

Learning from a Teacher using Unlabeled Data

Self-Training With Noisy Student Improves ImageNet Classification

PRADO: Projection Attention Networks for Document Classification On-Device

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Randaugment: Practical automated data augmentation with a reduced search space

Sparse Networks from Scratch: Faster Training without Losing Performance

AutoAugment: Learning Augmentation Strategies From Data

Transferable Neural Projection Representations

Searching for MobileNetV3

Billion-scale semi-supervised learning for image classification

An Attentive Survey of Attention Models

MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning

The State of Sparsity in Deep Neural Networks

FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search

Microsoft Research

Rethinking the Value of Network Pruning

SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation

MnasNet: Platform-Aware Neural Architecture Search for Mobile

Tune: A Research Platform for Distributed Model Selection and Training

MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning

DARTS: Differentiable Architecture Search

Quantizing deep convolutional networks for efficient inference: A whitepaper

Glow: Graph Lowering Compiler Techniques for Neural Networks

The Lottery Ticket Hypothesis: Training Pruned Neural Networks

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

Unsupervised Representation Learning by Predicting Image Rotations

Model compression via distillation and quantization

Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions

AMC: AutoML for Model Compression and Acceleration on Mobile Devices

Efficient Neural Architecture Search via Parameter Sharing

Regularized Evolution for Image Classifier Architecture Search

Universal Language Model Fine-tuning for Text Classification

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges

Data Augmentation by Pairing Samples for Images Classification

Advances in Pre-Training Distributed Word Representations

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Progressive Neural Architecture Search

Population Based Training of Neural Networks

mixup: Beyond Empirical Risk Minimization

To prune, or not to prune: exploring the efficacy of pruning for model compression

Google Vizier: A Service for Black-Box Optimization

ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections

Learning Transferable Architectures for Scalable Image Recognition

On Compressing Deep Models by Low Rank and Sparse Decomposition