An Empirical Model of Large-Batch Training (2018-12-14T00:00:00.000000Z)

TL;DR

It is demonstrated that a simple and easy-to-measure statistic called the gradient noise scale predicts the largest useful batch size across many domains and applications, including a number of supervised learning datasets, reinforcement learning domains, and even generative model training (autoencoders on SVHN).

Abstract

In an increasing number of domains it has been demonstrated that deep learning models can be trained using relatively large batch sizes without sacrificing data efficiency. However the limits of this massive data parallelism seem to differ from domain to domain, ranging from batches of tens of thousands in ImageNet to batches of millions in RL agents that play the game Dota 2. To our knowledge there is limited conceptual understanding of why these limits to batch size differ or how we might choose the correct batch size in a new domain. In this paper, we demonstrate that a simple and easy-to-measure statistic called the gradient noise scale predicts the largest useful batch size across many domains and applications, including a number of supervised learning datasets (MNIST, SVHN, CIFAR-10, ImageNet, Billion Word), reinforcement learning domains (Atari and Dota), and even generative model training (autoencoders on SVHN). We find that the noise scale increases as the loss decreases over a training run and depends on the model size primarily through improved model performance. Our empirically-motivated theory also describes the tradeoff between compute-efficiency and time-efficiency, and provides a rough model of the benefits of adaptive batch-size training.

Authors

Dario Amodei

10 papers

Sam McCandlish

7 papers

J. Kaplan

2 papers

TL;DR

Abstract

Authors

References60 items

Measuring the Effects of Data Parallelism on Neural Network Training

Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms?

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Gradient Descent Happens in a Tiny Subspace

On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent

Large Scale Language Modeling: Converging on 40GB of Text in Four Hours

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

The Effect of Network Width on the Performance of Large-batch Training

Scaling Neural Machine Translation

Accelerated Methods for Deep Reinforcement Learning

Understanding Short-Horizon Bias in Stochastic Meta-Optimization

Measuring the Intrinsic Dimension of Objective Landscapes

Distributed Prioritized Experience Replay

Distributed Deep Reinforcement Learning: Learn how to play Atari games in 21 minutes

The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning

AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks

Deep Learning Scaling is Predictable, Empirically

Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

Don't Decay the Learning Rate, Increase the Batch Size

A Bayesian Perspective on Generalization and Stochastic Gradient Descent

ImageNet Training in Minutes

Large Batch Training of Convolutional Networks

Proximal Policy Optimization Algorithms

Gradient Diversity: a Key Ingredient for Scalable Distributed Learning

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Train longer, generalize better: closing the generalization gap in large batch training of neural networks

Stochastic Gradient Descent as Approximate Bayesian Inference

Why Momentum Really Works

Opening the Black Box of Deep Neural Networks via Information

Coupling Adaptive Batch Sizes with Learning Rates

Distributed Second-Order Optimization using Kronecker-Factored Approximations

Big Batch SGD: Automated Inference using Adaptive Batch Sizes

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Optimization Methods for Large-Scale Machine Learning

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Asynchronous Methods for Deep Reinforcement Learning

Deep Residual Learning for Image Recognition

Adding Gradient Noise Improves Learning for Very Deep Networks

Neural Machine Translation of Rare Words with Subword Units

Adam: A Method for Stochastic Optimization

Qualitatively characterizing neural network optimization problems

Auto-Encoding Variational Bayes

One billion word benchmark for measuring progress in statistical language modeling

On the importance of initialization and momentum in deep learning

Sample size selection in optimization methods for machine learning

The Arcade Learning Environment: An Evaluation Platform for General Agents

No more pesky learning rates

ImageNet: A large-scale hierarchical image database

TrueSkillTM: A Bayesian Skill Rating System

Long Short-Term Memory

AI and Compute, May 2018

Curtis , and Jorge Nocedal

Sample Size Determination

Reading Digits in Natural Images with Unsupervised Feature Learning

MNIST handwritten digit database

Learning Multiple Layers of Features from Tiny Images

Selecting sample sizes

More on Dota 2

OpenAI Five

Tensorflow official models

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names