Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples (2021-11-04T00:00:00.000000Z)

TL;DR

Qimera, a method that uses superposed latent embeddings to generate synthetic boundary supporting samples and uses an additional disentanglement mapping layer and extracting information from the full-precision model to better reflect the original distribution is proposed.

Abstract

Model quantization is known as a promising method to compress deep neural networks, especially for inferences on lightweight mobile or edge devices. However, model quantization usually requires access to the original training data to maintain the accuracy of the full-precision models, which is often infeasible in real-world scenarios for security and privacy issues. A popular approach to perform quantization without access to the original data is to use synthetically generated samples, based on batch-normalization statistics or adversarial learning. However, the drawback of such approaches is that they primarily rely on random noise input to the generator to attain diversity of the synthetic samples. We find that this is often insufficient to capture the distribution of the original data, especially around the decision boundaries. To this end, we propose Qimera, a method that uses superposed latent embeddings to generate synthetic boundary supporting samples. For the superposed embeddings to better reflect the original distribution, we also propose using an additional disentanglement mapping layer and extracting information from the full-precision model. The experimental results show that Qimera achieves state-of-the-art performances for various settings on data-free quantization. Code is available at https://github.com/iamkanghyunchoi/qimera.

Authors

Kanghyun Choi

2 papers

Deokki Hong

2 papers

Noseong Park

5 papers

TL;DR

Abstract

Authors

References54 items

AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compression

Zero-shot Adversarial Quantization

Diversifying Sample Generation for Accurate Data-Free Quantization

Adversarial Metric Knowledge Distillation

Open-Set Interference Signal Recognition Using Boundary Samples: A Hybrid Approach

Data-Free Network Quantization With Adversarial Knowledge Distillation

Generative Low-bitwidth Data Free Quantization

Universal Domain Adaptation through Self Supervision

ZeroQ: A Novel Zero Shot Quantization Framework

Characterizing the Decision Boundary of Deep Neural Networks

Data-Free Adversarial Distillation

Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

Knowledge Extraction with No Observable Data

Data-Free Quantization Through Weight Equalization and Bias Correction

Zero-shot Knowledge Transfer via Adversarial Belief Matching

Zero-Shot Knowledge Distillation in Deep Networks

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

SinGAN: Learning a Generative Model From a Single Natural Image

Defensive Quantization: When Efficiency Meets Robustness

Semi-Supervised Domain Adaptation via Minimax Entropy

Data-Free Learning of Student Networks

Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet

Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

A Style-Based Generator Architecture for Generative Adversarial Networks

Post training 4-bit quantization of convolutional networks for rapid-deployment

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

MixUp as Locally Linear Out-Of-Manifold Regularization

Knowledge Distillation with Adversarial Samples Supporting Decision Boundary

Spectral Normalization for Generative Adversarial Networks

Feature-Based Metrics for Exploring the Latent Space of Generative Models

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Towards Effective Low-Bitwidth Convolutional Neural Networks

Latent Space Oddity: on the Curvature of Deep Generative Models

Automatic differentiation in PyTorch

mixup: Beyond Empirical Risk Minimization

Data-Free Knowledge Distillation for Deep Neural Networks

Modulating early visual processing by language

In-datacenter performance analysis of a tensor processing unit

Conditional Image Synthesis with Auxiliary Classifier GANs

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Deep Residual Learning for Image Recognition

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Distilling the Knowledge in a Neural Network

Adam: A Method for Stochastic Optimization

Conditional Generative Adversarial Nets

Intriguing properties of neural networks

ImageNet classification with deep convolutional neural networks

LIII. On lines and planes of closest fit to systems of points in space

GENERATIVE ADVERSARIAL NETS

Learning Multiple Layers of Features from Tiny Images

A method for solving the convex programming problem with convergence rate O(1/k^2)

Computer vision models on PyTorch

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names