Improving Neural Network Quantization without Retraining using Outlier Channel Splitting (2019-01-28T00:00:00.000000Z)

TL;DR

This work proposes outlier channel splitting (OCS), which duplicates channels containing outliers, then halves the channel values, and shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.

Abstract

Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS), which duplicates channels containing outliers, then halves the channel values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.

Authors

Ritchie Zhao

1 papers

Yuwei Hu

1 papers

Jordan Dotzel

1 papers

TL;DR

Abstract

Authors

References35 items

Improving Neural Network Quantization using Outlier Channel Splitting

Cell Division: Weight Bit-Width Reduction Technique for Convolutional Neural Network Hardware Accelerators

ACIQ: Analytical Clipping for Integer Quantization of neural networks

Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference

Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)

Energy-Efficient Neural Network Accelerator Based on Outlier-Aware Low-Precision Computation

Quantizing Convolutional Neural Networks for Low-Power High-Throughput Inference Engines

Value-aware Quantization for Training and Inference of Neural Networks

Serving DNNs in Real Time at Datacenter Scale with Project Brainwave

Scaling for edge inference of deep neural networks

PACT: Parameterized Clipping Activation for Quantized Neural Networks

Training and Inference with Integers in Deep Neural Networks

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Towards Effective Low-Bitwidth Convolutional Neural Networks

Automatic differentiation in PyTorch

In-datacenter performance analysis of a tensor processing unit

Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights

Pointer Sentinel Mixture Models

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

Densely Connected Convolutional Networks

Deep Residual Learning for Image Recognition

Fixed-point performance analysis of recurrent neural networks

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

Resiliency of Deep Neural Networks under Quantization

Net2Net: Accelerating Learning via Knowledge Transfer

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Going deeper with convolutions

Recurrent Neural Network Regularization

Very Deep Convolutional Networks for Large-Scale Image Recognition

Microsoft COCO: Common Objects in Context

ImageNet: A large-scale hierarchical image database

Inference with TensorRT

Learning Multiple Layers of Features from Tiny Images

Mathematical Miniatures , chapter 12

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names