On the Variance of the Adaptive Learning Rate and Beyond (2019-08-08T00:00:00.000000Z)

TL;DR

This work identifies a problem of the adaptive learning rate, suggests warmup works as a variance reduction technique, and proposes RAdam, a new variant of Adam, by introducing a term to rectify the variance of theadaptive learning rate.

Abstract

The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Extensive experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the effectiveness and robustness of our proposed method. All implementations are available at: this https URL.

Authors

Jianfeng Gao

39 papers

Weizhu Chen

13 papers

Liyuan Liu

3 papers

TL;DR

Abstract

Authors

References39 items

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Reliability-aware Dynamic Feature Composition for Name Tagging

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Adaptive Gradient Methods with Dynamic Bound of Learning Rate

Fixup Initialization: Residual Learning Without Normalization

A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks

Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling

Training Tips for the Transformer Model

On the Convergence of Adam and Beyond

signSGD: compressed optimisation for non-convex problems

Decoupled Weight Decay Regularization

Fixing Weight Decay Regularization in Adam

DSCOVR: Randomized Primal-Dual Block Coordinate Algorithms for Asynchronous Distributed Optimization

Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks

Attention is All you Need

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

The Shattered Gradients Problem: If resnets are the answer, then what is the question?

Layer Normalization

Incorporating Nesterov Momentum into Adam

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

One billion word benchmark for measuring progress in statistical language modeling

ADADELTA: An Adaptive Learning Rate Method

Advances in optimizing recurrent networks

The conference paper

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

ImageNet: A large-scale hierarchical image database

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Forecasting with moving averages

Report on the 11th IWSLT evaluation campaign

Neural networks for machine learning lecture 6a overview of mini-batch gradient descent

Learning Multiple Layers of Features from Tiny Images

Taylor Series Methods

Theoria combinationis observationum erroribus minimis obnoxiae

Méthode générale pour la résolution des systemes d'équations simultanées

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names