Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition (2021-07-23T00:00:00.000000Z)

Abstract

We present a novel global compression framework for deep neural networks that automatically analyzes each layer to identify the optimal per-layer compression ratio, while simultaneously achieving the desired overall compression. Our algorithm hinges on the idea of compressing each convolutional (or fully-connected) layer by slicing its channels into multiple groups and decomposing each group via low-rank decomposition. At the core of our algorithm is the derivation of layer-wise error bounds from the Eckart Young Mirsky theorem. We then leverage these bounds to frame the compression problem as an optimization problem where we wish to minimize the maximum compression error across layers and propose an efficient algorithm towards a solution. Our experiments indicate that our method outperforms existing low-rank compression approaches across a wide range of networks and data sets. We believe that our results open up new avenues for future research into the global performance-size trade-offs of modern neural networks. Our code is available at https://github.com/lucaslie/torchprune.

Authors

Lucas Liebenwein

1 papers

Alaa Maalouf

1 papers

O. Gal

1 papers

Abstract

Authors

References110 items

Sparse Flows: Pruning Continuous-depth Models

Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy

ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations

Neural Pruning via Growing Regularization

Deep Learning Meets Projective Clustering

Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation

Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer

TRP: Trained Rank Pruning for Efficient Deep Neural Networks

Dynamic Model Pruning with Feedback

WoodFisher: Efficient second-order approximations for model compression

HRank: Filter Pruning Using High-Rank Feature Map

Provable Filter Pruning for Efficient Neural Networks

SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks

Automated Multi-Stage Compression of Neural Networks

Learning Filter Basis for Convolutional Neural Network Compression

Importance Estimation for Neural Network Pruning

OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networks

Approximated Oracle Filter Pruning for Destructive CNN Width Optimization

Towards Efficient Model Compression via Learned Global Ranking

MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning

Efficient Neural Network Compression

Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration

Discrimination-aware Channel Pruning for Deep Neural Networks

Dynamic Channel Pruning: Feature Boosting and Suppression

Rethinking the Value of Network Pruning

Constrained Optimization Based Low-Rank Approximation of Deep Neural Networks

Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks

GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking

Channel Gating Neural Networks

AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference

Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds

Stronger generalization bounds for deep nets via a compression approach

Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers

MobileNetV2: Inverted Residuals and Linear Bottlenecks

NISP: Pruning Networks Using Neuron Importance Score Propagation

Compression-aware Training of Deep Networks

Automatic differentiation in PyTorch

Learning Efficient Convolutional Networks through Network Slimming

On Compressing Deep Models by Low Rank and Sparse Decomposition

Channel Pruning for Accelerating Very Deep Neural Networks

Rethinking Atrous Convolution for Semantic Image Segmentation

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Coordinating Filters for Faster Deep Neural Networks

More is Less: A More Complicated Network with Less Inference Complexity

Soft Weight-Sharing for Neural Network Compression

Ultimate tensorization: compressing convolutional and FC layers alike

Pruning Convolutional Neural Networks for Resource Efficient Inference

Pruning Filters for Efficient ConvNets

Densely Connected Convolutional Networks

Wide Residual Networks

Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups

Quantized Convolutional Neural Networks for Mobile Devices

Deep Residual Learning for Image Recognition

Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications

Training CNNs with Low-Rank Filters for Efficient Image Classification

Convolutional neural networks with low-rank regularization

Input Sparsity and Hardness for Robust Subspace Approximation

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Tensorizing Neural Networks

Accelerating Very Deep Convolutional Networks for Classification and Detection

Dimensionality Reduction via Regression in Hyperspectral Imagery

Compressing Neural Networks with the Hashing Trick

Distilling the Knowledge in a Neural Network

Robust Subspace Clustering via Thresholding Ridge Regression

Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition

Efficient and accurate approximations of nonlinear convolutional networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

The Pascal Visual Object Classes Challenge: A Retrospective

Speeding up Convolutional Neural Networks with Low Rank Expansions

Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

Predicting Parameters in Deep Learning

Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets

ImageNet classification with deep convolutional neural networks

Semantic contours from inverse detectors

Hash Kernels for Structured Data

Feature hashing for large scale multitask learning