The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018-03-09T00:00:00.000000Z)

TL;DR

This work finds that dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations, and articulate the "lottery ticket hypothesis".

Abstract

Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance. We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.

Authors

Jonathan Frankle

4 papers

Michael Carbin

4 papers

TL;DR

Abstract

Authors

References68 items

Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Rethinking the Value of Network Pruning

Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach

Compressibility and Generalization in Large-Scale Deep Learning

Measuring the Intrinsic Dimension of Objective Landscapes

Stronger generalization bounds for deep nets via a compression approach

Learning Sparse Neural Networks through L0 Regularization

Deep Rewiring: Training very sparse deep networks

ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression

Channel Pruning for Accelerating Very Deep Neural Networks

A Closer Look at Memorization in Deep Networks

Bayesian Compression for Deep Learning

Concrete Dropout

Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

Structured Bayesian Pruning via Log-Normal Multiplicative Noise

Exploring Sparsity in Recurrent Neural Networks

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Variational Dropout Sparsifies Deep Neural Networks

Training Sparse Neural Networks

Generalized Dropout

Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning

Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning

Understanding deep learning requires rethinking generalization

Pruning Convolutional Neural Networks for Resource Efficient Inference

Pruning Filters for Efficient ConvNets

Towards Evaluating the Robustness of Neural Networks

Dynamic Network Surgery for Efficient DNNs

Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods

DSD: Regularizing Deep Neural Networks with Dense-Sparse-Dense Training Flow

Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

Inductive Bias of Deep Convolutional Networks through Pooling Geometry

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size

RandomOut: Using a convolutional gradient norm to win The Filter Lottery

Deep Residual Learning for Image Recognition

Learning Neural Network Architectures using Backpropagation

Learning the Architecture of Deep Neural Networks

Diversity Networks

Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks

Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding

Data-free Parameter Pruning for Deep Neural Networks

Learning both Weights and Connections for Efficient Neural Network

Variational Dropout and the Local Reparameterization Trick

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Distilling the Knowledge in a Neural Network

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Do Deep Nets Really Need to be Deep?

Understanding Dropout

Regularization of Neural Networks using DropConnect

Predicting Parameters in Deep Learning

Improving neural networks by preventing co-adaptation of feature detectors

Understanding the difficulty of training deep feedforward neural networks

Convex Neural Networks

Optimal Brain Surgeon and general network pruning

Second Order Derivatives for Network Pruning: Optimal Brain Surgeon

Stochastic Complexity and Modeling

Occam's razor

The resnet18 architecture

Under review as a conference

2016)) never lead to higher accuracy as the network is pruned

2016), this third stage likely learns at too slow of a rate to tangibly influence validation accuracy

Dropout: a simple way to prevent neural networks from overfitting

Learning Multiple Layers of Features from Tiny Images

Gradient-based learning applied to document recognition

Optimal Brain Damage

Field of Study

Journal Information

Name

Venue Information

Name

Type

URL

Alternate Names