Soft Weight-Sharing for Neural Network Compression (2017-02-13T00:00:00.000000Z)

TL;DR

This paper shows that competitive compression rates can be achieved by using a version of “soft weight-sharing” (Nowlan & Hinton, 1992) and achieves both quantization and pruning in one simple (re-)training procedure, exposing the relation between compression and the minimum description length (MDL) principle.

Abstract

The success of deep learning in numerous application domains created the desire to run and train them on mobile devices. This however, conflicts with their computationally, memory and energy intense nature, leading to a growing interest in compression. Recent work by Han et al. (2015a) propose a pipeline that involves retraining, pruning and quantization of neural network weights, obtaining state-of-the-art compression rates. In this paper, we show that competitive compression rates can be achieved by using a version of “soft weight-sharing” (Nowlan & Hinton, 1992). Our method achieves both quantization and pruning in one simple (re-)training procedure. This point of view also exposes the relation between compression and the minimum description length (MDL) principle.

Authors

M. Welling

17 papers

Karen Ullrich

3 papers

Edward Meeds

1 papers

TL;DR

Abstract

Authors

References47 items

Variational Dropout Sparsifies Deep Neural Networks

Accelerating Deep Convolutional Networks using low-precision and sparsity

Dynamic Network Surgery for Efficient DNNs

Learning Structured Sparsity in Deep Neural Networks

Deep neural networks are robust to weight binarization and other non-linear distortions

Wide Residual Networks

Identity Mappings in Deep Residual Networks

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size

BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

Binarized Neural Networks

Quantized Convolutional Neural Networks for Mobile Devices

Deep Residual Learning for Image Recognition

Training CNNs with Low-Rank Filters for Efficient Image Classification

Fixed Point Quantization of Deep Convolutional Networks

Convolutional neural networks with low-rank regularization

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Learning both Weights and Connections for Efficient Neural Network

Sparse Convolutional Neural Networks

Distilling the Knowledge in a Neural Network

Deep Learning with Limited Numerical Precision

Adam: A Method for Stochastic Optimization

Training deep neural networks with low precision multiplications

Compressing Deep Convolutional Networks using Vector Quantization

Speeding up Convolutional Neural Networks with Low Rank Expansions

Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

Predicting Parameters in Deep Learning

Multiresolution Mixture Modeling using Merging of Mixture Components

Practical Bayesian Optimization of Machine Learning Algorithms

The conference paper

Practical Variational Inference for Neural Networks

Variational learning and bits-back coding: an information-theoretic view to Bayesian learning

Keeping the neural networks simple by minimizing the description length of the weights

Simplifying Neural Networks by Soft Weight-Sharing

Classification by Minimum-Message-Length Inference

Arithmetic coding for data compression

Stochastic Complexity and Modeling

Modeling By Shortest Data Description*

A Stochastic Approximation Method

Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS

Under Review as a Conference Paper at Iclr 2017 Delving into Transferable Adversarial Ex- Amples and Black-box Attacks

Gradient-based learning applied to document recognition

(b) LeNet-5-Caffe a modiﬁed version of the LeNet-5 MNIST model in LeCun et al.

(a) LeNet-300-100 an MNIST model described in LeCun et al.

Optimal Brain Damage

Pattern recognition

we show the pre-trained and compressed ﬁlters for the ﬁrst and second layers of LeNet-5-Caffe. For some of the feature maps from layer 2 seem to be redundant hence the almost empty columns

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names