ZeroQ: A Novel Zero Shot Quantization Framework (2020-01-02T00:00:00.000000Z)

TL;DR

THE AUTHORS' enables mixed-precision quantization without any access to the training or validation data, and it can finish the entire quantization process in less than 30s, which is very low computational overhead.

Abstract

Quantization is a promising approach for reducing the inference time and memory footprint of neural networks. However, most existing quantization methods require access to the original training dataset for retraining during quantization. This is often not possible for applications with sensitive or proprietary data, e.g., due to privacy and security concerns. Existing zero-shot quantization methods use different heuristics to address this, but they result in poor performance, especially when quantizing to ultra-low precision. Here, we propose \OURS, a novel zero-shot quantization framework to address this. \OURS enables mixed-precision quantization without any access to the training or validation data. This is achieved by optimizing for a Distilled Dataset, which is engineered to match the statistics of batch normalization across different layers of the network. \OURS supports both uniform and mixed-precision quantization. For the latter, we introduce a novel Pareto frontier based method to automatically determine the mixed-precision bit setting for all layers, with no manual search involved. We extensively test our proposed method on a diverse set of models, including ResNet18/50/152, MobileNetV2, ShuffleNet, SqueezeNext, and InceptionV3 on ImageNet, as well as RetinaNet-ResNet50 on the Microsoft COCO dataset. In particular, we show that \OURS can achieve 1.71\% higher accuracy on MobileNetV2, as compared to the recently proposed DFQ~\cite{nagel2019data} method. Importantly, \OURS has a very low computational overhead, and it can finish the entire quantization process in less than 30s (0.5\% of one epoch training time of ResNet50 on ImageNet). We have open-sourced the \OURS framework\footnote{https://github.com/amirgholami/ZeroQ}.

Authors

References48 items

The Knowledge Within: Methods for Data-Free Model Compression

HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

CAT: Compression-Aware Training for bandwidth reduction

Co-design of deep neural nets and neural net accelerators for embedded vision applications

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

Data-Free Quantization Through Weight Equalization and Bias Correction

Fully Quantized Network for Object Detection

HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision

Low-bit Quantization of Neural Networks for Efficient Inference

Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization

Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

HAQ: Hardware-Aware Automated Quantization

Post training 4-bit quantization of convolutional networks for rapid-deployment

Quantization for Rapid Deployment of Deep Neural Networks

Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search

LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks

Quantizing deep convolutional networks for efficient inference: A whitepaper

Value-aware Quantization for Training and Inference of Neural Networks

Invited: Co-Design of Deep Neural Nets and Neural Net Accelerators for Embedded Vision Applications

SqueezeNext: Hardware-Aware Neural Network Design

PACT: Parameterized Clipping Activation for Quantized Neural Networks

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Adaptive Quantization for Deep Neural Network

Focal Loss for Dense Object Detection

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

Exploring the Regularity of Sparse Structure in Convolutional Neural Networks

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights

Feature Pyramid Networks for Object Detection

Trained Ternary Quantization

Pruning Filters for Efficient ConvNets

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size

Quantized Convolutional Neural Networks for Mobile Devices

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Learning both Weights and Connections for Efficient Neural Network

Distilling the Knowledge in a Neural Network

FitNets: Hints for Thin Deep Nets

Microsoft COCO: Common Objects in Context

1.1 Computing's energy problem (and what we can do about it)

- LEVEL ACCURACY WITH 50 X FEWER PARAMETERS AND < 0 . 5 MB MODEL SIZE

Inceptionism: Going Deeper into Neural Networks

Experimental Determination of Precision Requirements for Back-propagation Training of Artiicial Neural Networks