SGDR: Stochastic Gradient Descent with Warm Restarts
Published in
International Conference on Learning Representa...(2016)External Links:
TL
TL;DR
This paper proposes a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks and empirically studies its performance on the CIFAR-10 and CIFARS datasets.
Abstract
Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at this https URL
References42 items
1
Snapshot Ensembles: Train 1, get M for free
2
Deep Pyramidal Residual Networks
3
Densely Connected Convolutional Networks
4
Residual Networks of Residual Networks: Multilevel Residual Networks
5