Diverse Sample Generation: Pushing the Limit of Generative Data-Free Quantization (2021-09-01T00:00:00.000000Z)

TL;DR

This paper presents a generic Diverse Sample Generation (DSG) scheme for the generative data-free quantization, to mitigate detrimental homogenization and strengthens the loss impact of the specific BN layers for different samples and inhibit the correlation among samples in the generation process.

Abstract

Generative data-free quantization emerges as a practical compression approach that quantizes deep neural networks to low bit-width without accessing the real data. This approach generates data utilizing batch normalization (BN) statistics of the full-precision networks to quantize the networks. However, it always faces the serious challenges of accuracy degradation in practice. We first give a theoretical analysis that the diversity of synthetic samples is crucial for the data-free quantization, while in existing approaches, the synthetic data completely constrained by BN statistics experimentally exhibit severe homogenization at distribution and sample levels. This paper presents a generic Diverse Sample Generation (DSG) scheme for the generative data-free quantization, to mitigate detrimental homogenization. We first slack the statistics alignment for features in the BN layer to relax the distribution constraint. Then, we strengthen the loss impact of the specific BN layers for different samples and inhibit the correlation among samples in the generation process, to diversify samples from the statistical and spatial perspectives, respectively. Comprehensive experiments show that for large-scale image classification tasks, our DSG can consistently quantization performance on different neural architectures, especially under ultra-low bit-width. And data diversification caused by our DSG brings a general gain to various quantization-aware training and post-training quantization approaches, demonstrating its generality and effectiveness.

Authors

Jiwen Lu

18 papers

Xianglong Liu

7 papers

Haotong Qin

2 papers

TL;DR

Abstract

Authors

References64 items

SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation

Zero-Shot Learning Of A Conditional Generative Adversarial Network For Data-Free Network Quantization

Zero-shot Adversarial Quantization

Diversifying Sample Generation for Accurate Data-Free Quantization

BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

Generative Zero-shot Network Quantization

Learning Efficient Binarized Object Detectors With Information Compression

BiPointNet: Binary Neural Network for Point Clouds

Towards Efficient U-Nets: A Coupled and Quantized Approach

Data-Free Network Quantization With Adversarial Knowledge Distillation

Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond

Up or Down? Adaptive Rounding for Post-Training Quantization

Binary Neural Networks: A Survey

Generative Low-bitwidth Data Free Quantization

Deep Neural Network Compression by In-Parallel Pruning-Quantization

Discrimination-Aware Network Pruning for Deep Model Compression

ZeroQ: A Novel Zero Shot Quantization Framework

Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion

The Knowledge Within: Methods for Data-Free Model Compression

A survey on semi-supervised learning

Forward and Backward Information Retention for Accurate Binary Neural Networks

Effective Training of Convolutional Neural Networks With Low-Bitwidth Weights and Activations

Learning Deep Binary Descriptor with Multi-Quantization

Data-Free Quantization Through Weight Equalization and Bias Correction

Learning Channel-Wise Interactions for Binary Convolutional Neural Networks

MixMatch: A Holistic Approach to Semi-Supervised Learning

Libra R-CNN: Towards Balanced Learning for Object Detection

Implicit Kernel Learning

Low-bit Quantization of Neural Networks for Efficient Inference

Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

GDPP: Learning Diverse Generations Using Determinantal Point Process

Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation

Post training 4-bit quantization of convolutional networks for rapid-deployment

Kernel Change-point Detection with Auxiliary Deep Generative Models

Hierarchical Binary CNNs for Landmark Localization with Limited Resources

Value-aware Quantization for Training and Inference of Neural Networks

SqueezeNext: Hardware-Aware Neural Network Design

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

MMD GAN: Towards Deeper Understanding of Moment Matching Network

Conditional Image Synthesis with Auxiliary Classifier GANs

Determinantal Point Processes

Deep Residual Learning for Image Recognition

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

Rethinking the Inception Architecture for Computer Vision

Resiliency of Deep Neural Networks under Quantization

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Fast R-CNN

Deep Learning with Limited Numerical Precision

Very Deep Convolutional Networks for Large-Scale Image Recognition

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

ImageNet classification with deep convolutional neural networks

Determinantal Point Processes for Machine Learning

ImageNet: A large-scale hierarchical image database

Kernel Learning by Unconstrained Optimization

PRINCIPAL COMPONENT ANALYSIS AND FACTOR ANALYSIS

Matrix analysis

Matrix of ones — Wikipedia, the free Encyclopedia

Inceptionism: Going Deeper into Neural Networks

Learning Multiple Layers of Features from Tiny Images

The PASCAL Visual Object Classes Challenge

Semi-Supervised Learning , Cam-bridge, MA, USA

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names