A flexible, extensible software framework for model compression based on the LC algorithm (2020-05-15T00:00:00.000000Z)

TL;DR

A software framework that allows a user to compress a neural network or other machine learning model using different compression schemes with minimal effort, and the compressed model is competitive in terms of prediction accuracy and compression ratio with other algorithms.

Abstract

We propose a software framework based on the ideas of the Learning-Compression (LC) algorithm, that allows a user to compress a neural network or other machine learning model using different compression schemes with minimal effort. Currently, the supported compressions include pruning, quantization, low-rank methods (including automatically learning the layer ranks), and combinations of those, and the user can choose different compression types for different parts of a neural network. The LC algorithm alternates two types of steps until convergence: a learning (L) step, which trains a model on a dataset (using an algorithm such as SGD); and a compression (C) step, which compresses the model parameters (using a compression scheme such as low-rank or quantization). This decoupling of the "machine learning" aspect from the "signal compression" aspect means that changing the model or the compression type amounts to calling the corresponding subroutine in the L or C step, respectively. The library fully supports this by design, which makes it flexible and extensible. This does not come at the expense of performance: the runtime needed to compress a model is comparable to that of training the model in the first place; and the compressed model is competitive in terms of prediction accuracy and compression ratio with other algorithms (which are often specialized for specific models or compression schemes). The library is written in Python and PyTorch and available in Github.

Authors

Yerlan Idelbayev

3 papers

Miguel 'A. Carreira-Perpin'an

2 papers

TL;DR

Abstract

Authors

References60 items

Optimal Selection of Matrix Shape and Decomposition Scheme for Neural Network Compression

Neural Network Compression via Additive Combination of Reshaped, Low-Rank Matrices

Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer

TRP: Trained Rank Pruning for Efficient Deep Neural Networks

Neural Network Compression Framework for fast model inference

Neural Network Distiller: A Python Package For DNN Compression Research

Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-Based Approach

OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networks

Approximated Oracle Filter Pruning for Destructive CNN Width Optimization

Trained Rank Pruning for Efficient Deep Neural Networks

Efficient Neural Network Compression

Discrimination-aware Channel Pruning for Deep Neural Networks

Constrained Optimization Based Low-Rank Approximation of Deep Neural Networks

"Learning-Compression" Algorithms for Neural Net Pruning

CLIP-Q: Deep Network Compression Learning by In-parallel Pruning-Quantization

Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

NISP: Pruning Networks Using Neuron Importance Score Propagation

Channel Pruning for Accelerating Very Deep Neural Networks

Model compression as constrained optimization, with application to neural nets. Part II: quantization

Model compression as constrained optimization, with application to neural nets. Part I: general framework

Coordinating Filters for Faster Deep Neural Networks

Ultimate tensorization: compressing convolutional and FC layers alike

Trained Ternary Quantization

Towards the Limit of Network Quantization

Pruning Filters for Efficient ConvNets

Learning Structured Sparsity in Deep Neural Networks

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Quantized Convolutional Neural Networks for Mobile Devices

Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications

Convolutional neural networks with low-rank regularization

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Tensorizing Neural Networks

Learning both Weights and Connections for Efficient Neural Network

Sparse Convolutional Neural Networks

Accelerating Very Deep Convolutional Networks for Classification and Detection

Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition

Compressing Deep Convolutional Networks using Vector Quantization

Fixed-point feedforward deep neural network design using weights +1, 0, and −1

Caffe: Convolutional Architecture for Fast Feature Embedding

Speeding up Convolutional Neural Networks with Low Rank Expansions

Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

Predicting Parameters in Deep Learning

Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets

Random Projection Trees for Vector Quantization

NP-hardness of Euclidean sum-of-squares clustering

Second Order Derivatives for Network Pruning: Optimal Brain Surgeon

Optimal Quantization by Matrix Searching

An O(KN lg N) algorithm for optimum K-level quantization on histograms of N points

A universal algorithm for sequential data compression

On optimum quantization

More General and Effective Model Compression via an Additive Combination of Compressions

A flexible, extensible software framework for model compression based on the LC algorithm

Model compression as constrained optimization, with application to neural nets

Restructuring of deep neural network acoustic models with singular value decomposition

Optimal Brain Damage

L steps

For the base compressions in the framework, we made sure they always optimize the quadratic distortion loss of the C step

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names