SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation (2022-02-14T00:00:00.000000Z)

TL;DR

This paper proposes an on-the-fly DFQ framework with sub-second quantization time, called SQuant, which can quantize networks on inference-only devices with low computation and memory requirements, and proposes a novel data-free optimization objective in the discrete domain, minimizing Constrained Absolute Sum of Error.

Abstract

Quantization of deep neural networks (DNN) has been proven effective for compressing and accelerating DNN models. Data-free quantization (DFQ) is a promising approach without the original datasets under privacy-sensitive and confidential scenarios. However, current DFQ solutions degrade accuracy, need synthetic data to calibrate networks, and are time-consuming and costly. This paper proposes an on-the-fly DFQ framework with sub-second quantization time, called SQuant, which can quantize networks on inference-only devices with low computation and memory requirements. With the theoretical analysis of the second-order information of DNN task loss, we decompose and approximate the Hessian-based optimization objective into three diagonal sub-items, which have different areas corresponding to three dimensions of weight tensor: element-wise, kernel-wise, and output channel-wise. Then, we progressively compose sub-items and propose a novel data-free optimization objective in the discrete domain, minimizing Constrained Absolute Sum of Error (or CASE in short), which surprisingly does not need any dataset and is even not aware of network architecture. We also design an efficient algorithm without back-propagation to further reduce the computation complexity of the objective solver. Finally, without fine-tuning and synthetic datasets, SQuant accelerates the data-free quantization process to a sub-second level with>30% accuracy improvement over the existing data-free post-training quantization works, with the evaluated models under 4-bit quantization. We have open-sourced the SQuant framework at https://github.com/clevercool/SQuant.

Authors

References46 items

Block-Skim: Efficient Question Answering for Transformer

Dual-side Sparse Tensor Core

Zero-shot Adversarial Quantization

Diversifying Sample Generation for Accurate Data-Free Quantization

BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

Hessian-Aware Pruning and Optimal Neural Implant

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

How Far Does BERT Look At: Distance-based Clustering and Analysis of BERT’s Attention

Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks

Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity

Channel-wise Hessian Aware trace-Weighted Quantization of Neural Networks

Sparse GPU Kernels for Deep Learning

Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming

Data-Free Network Quantization With Adversarial Knowledge Distillation

Up or Down? Adaptive Rounding for Post-Training Quantization

Generative Low-bitwidth Data Free Quantization

Balancing Efficiency and Flexibility for DNN Acceleration via Temporal GPU-Systolic Array Integration

SpArch: Efficient Architecture for Sparse Matrix Multiplication

ZeroQ: A Novel Zero Shot Quantization Framework

PyTorch: An Imperative Style, High-Performance Deep Learning Library

HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

SparTen: A Sparse Tensor Accelerator for Convolutional Neural Networks

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

Effective Training of Convolutional Neural Networks With Low-Bitwidth Weights and Activations

Data-Free Quantization Through Weight Equalization and Bias Correction

Learning Channel-Wise Interactions for Binary Convolutional Neural Networks

HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision

Adversarial Defense Through Network Profiling Based Path Extraction

Low-bit Quantization of Neural Networks for Efficient Inference

Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

Post training 4-bit quantization of convolutional networks for rapid-deployment

SqueezeNext: Hardware-Aware Neural Network Design

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Learning both Weights and Connections for Efficient Neural Network

Deep Learning with Limited Numerical Precision

ImageNet classification with deep convolutional neural networks

Diverse Sample Generation: Pushing the Limit of Data-free Quantization

HAWQ-V3: Dyadic Neural Network Quantization

2 PROOF OF TOK-k PERTURBATION ALGORITHM

We will prove the SQuant-E&K will lead to the top-k algorithm. We have the composition SQuant-E and SQuant-K optimization objective for n-th kernel

( M · 2 N ) for