Model compression as constrained optimization, with application to neural nets. Part V: combining compressions (2021-07-09T00:00:00.000000Z)

TL;DR

experimentally with deep neural nets, it is observed that 1) the authors can find significantly better models in the error-compression space, indicating that different compression types have complementary benefits, and 2) the best type of combination depends exquisitely on the type of neural net.

Abstract

Model compression is generally performed by using quantization, low-rank approximation or pruning, for which various algorithms have been researched in recent years. One fundamental question is: what types of compression work better for a given model? Or even better: can we improve by combining compressions in a suitable way? We formulate this generally as a problem of optimizing the loss but where the weights are constrained to equal an additive combination of separately compressed parts; and we give an algorithm to learn the corresponding parts' parameters. Experimentally with deep neural nets, we observe that 1) we can find significantly better models in the error-compression space, indicating that different compression types have complementary benefits, and 2) the best type of combination depends exquisitely on the type of neural net. For example, we can compress ResNets and AlexNet using only 1 bit per weight without error degradation at the cost of adding a few floating point weights. However, VGG nets can be better compressed by combining low-rank with a few floating point weights.

Authors

Yerlan Idelbayev

3 papers

Miguel 'A. Carreira-Perpin'an

2 papers

TL;DR

Abstract

Authors

References59 items

Nonlinear Programming

Optimal Selection of Matrix Shape and Decomposition Scheme for Neural Network Compression

Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer

A flexible, extensible software framework for model compression based on the LC algorithm

TRP: Trained Rank Pruning for Efficient Deep Neural Networks

Adaptive Loss-Aware Quantization for Multi-Bit Networks

Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-Based Approach

Quantization Networks

OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networks

Approximated Oracle Filter Pruning for Destructive CNN Width Optimization

Efficient Neural Network Compression

Rethinking the Value of Network Pruning

Numerical Optimization

"Learning-Compression" Algorithms for Neural Net Pruning

CLIP-Q: Deep Network Compression Learning by In-parallel Pruning-Quantization

Deep Neural Network Compression with Single and Multiple Level Quantization

Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers

BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights

NISP: Pruning Networks Using Neuron Importance Score Propagation

Compression-aware Training of Deep Networks

Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM

On Compressing Deep Models by Low Rank and Sparse Decomposition

Model compression as constrained optimization, with application to neural nets. Part I: general framework

Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations

Coordinating Filters for Faster Deep Neural Networks

Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights

Trained Ternary Quantization

Towards the Limit of Network Quantization

Pruning Filters for Efficient ConvNets

Learning Structured Sparsity in Deep Neural Networks

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Ternary Weight Networks

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Quantized Convolutional Neural Networks for Mobile Devices

Deep Residual Learning for Image Recognition

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Tensorizing Neural Networks

Coordinate descent algorithms

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Additive Quantization for Extreme Vector Compression

Caffe: Convolutional Architecture for Fast Feature Embedding

On the Convergence of Block Coordinate Descent Type Methods

ImageNet classification with deep convolutional neural networks

GoDec: Randomized Lowrank & Sparse Matrix Decomposition in Noisy Case

Robust principal component analysis?

Rank-Sparsity Incoherence for Matrix Decomposition

Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization

Simplifying Neural Networks by Soft Weight-Sharing

Generalized Additive Models

More General and Effective Model Compression via an Additive Combination of Compressions

AutoPrune: Automatic Network Pruning by Regularizing Auxiliary Parameters

Model compression as constrained optimization, with application to neural nets

Optimal Brain Damage

A method for solving the convex programming problem with convergence rate O(1/k^2)

low-rank (Q+L) achieves good performance on the ResNet110, having compressed models outperforming the reference. However

Training compressed models We run our algorithm for 50 LC iterations on the ResNet-20/32, and for 45 iterations on the ResNet-56/110, with µ = × 10 − 3 × 1 . 1 k at k -th step

Reference model We train a multinomial logistic regression classiﬁer on the CIFAR10 dataset (60k color images of 32 × 32 pixels, 10 classes).

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names