Patch Similarity Aware Data-Free Quantization for Vision Transformers (2022-03-04T00:00:00.000000Z)

TL;DR

This paper proposes PSAQ-ViT, a Patch Similarity Aware data-free Quantization framework for Vision Transformers, to enable the generation of "realistic" samples based on the vision transformer's unique properties for calibrating the quantization parameters.

Abstract

Vision transformers have recently gained great success on various computer vision tasks; nevertheless, their high model complexity makes it challenging to deploy on resource-constrained devices. Quantization is an effective approach to reduce model complexity, and data-free quantization, which can address data privacy and security concerns during model deployment, has received widespread interest. Unfortunately, all existing methods, such as BN regularization, were designed for convolutional neural networks and cannot be applied to vision transformers with significantly different model architectures. In this paper, we propose PSAQ-ViT, a Patch Similarity Aware data-free Quantization framework for Vision Transformers, to enable the generation of"realistic"samples based on the vision transformer's unique properties for calibrating the quantization parameters. Specifically, we analyze the self-attention module's properties and reveal a general difference (patch similarity) in its processing of Gaussian noise and real images. The above insights guide us to design a relative value metric to optimize the Gaussian noise to approximate the real images, which are then utilized to calibrate the quantization parameters. Extensive experiments and ablation studies are conducted on various benchmarks to validate the effectiveness of PSAQ-ViT, which can even outperform the real-data-driven methods. Code is available at: https://github.com/zkkli/PSAQ-ViT.

Authors

Zhikai Li

2 papers

Liping Ma

1 papers

Mengjuan Chen

2 papers

TL;DR

Abstract

Authors

References42 items

I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference

PTQ4ViT: Post-training Quantization for Vision Transformers with Twin Uniform Quantization

IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantization

Rethinking and Improving Relative Position Encoding for Vision Transformer

Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation

Post-Training Quantization for Vision Transformer

Patch Slimming for Efficient Vision Transformers

Transformer Tracking

ViViT: A Video Vision Transformer

A Survey of Quantization Methods for Efficient Neural Network Inference

DeepViT: Towards Deeper Vision Transformer

Diversifying Sample Generation for Accurate Data-Free Quantization

Transformer in Transformer

BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

Video Transformer Network

Transformers in Vision: A Survey

Training data-efficient image transformers & distillation through attention

Pre-Trained Image Processing Transformer

Deformable DETR: Deformable Transformers for End-to-End Object Detection

One Weight Bitwidth to Rule Them All

EasyQuant: Post-training Quantization via Scale Optimization

End-to-End Object Detection with Transformers

Up or Down? Adaptive Rounding for Post-Training Quantization

Generative Low-bitwidth Data Free Quantization

Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization

ZeroQ: A Novel Zero Shot Quantization Framework

Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion

Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks

Fully Quantized Network for Object Detection

Learned Step Size Quantization

Low-bit Quantization of Neural Networks for Efficient Inference

LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks

Quantizing deep convolutional networks for efficient inference: A whitepaper

PACT: Parameterized Clipping Activation for Quantized Neural Networks

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

ImageNet classification with deep convolutional neural networks

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

FQ-ViT: Fully Quantized Vision Transformer without Retraining

Binarized Neural Networks

Patch Similarity Aware Data-Free Quantization for Vision

Field of Study

Venue Information

Name

Type

URL

Alternate Names