ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning (2020-06-01T00:00:00.000000Z)

TL;DR

ADAHESSIAN is a new stochastic optimization algorithm that directly incorporates approximate curvature information from the loss function, and it includes several novel performance-improving features, including a fast Hutchinson based method to approximate the curvature matrix with low computational overhead.

Abstract

Incorporating second-order curvature information into machine learning optimization algorithms can be subtle, and doing so naïvely can lead to high per-iteration costs associated with forming the Hessian and performing the associated linear system solve. To address this, we introduce ADAHESSIAN, a new stochastic optimization algorithm. ADAHESSIAN directly incorporates approximate curvature information from the loss function, and it includes several novel performance-improving features, including: (i) a fast Hutchinson based method to approximate the curvature matrix with low computational overhead; (ii) a spatial averaging to reduce the variance of the second derivative; and (iii) a root-mean-square exponential moving average to smooth out variations of the second-derivative across different iterations. We perform extensive tests on NLP, CV, and recommendation system tasks, and ADAHESSIAN achieves state-of-the-art results. In particular, we find that ADAHESSIAN: (i) outperforms AdamW for transformers by0.13/0.33 BLEU score on IWSLT14/WMT14, 2.7/1.0 PPLon PTB/Wikitext-103; (ii) outperforms AdamW for Squeeze-Bert by 0.41 points on GLUE; (iii) achieves 1.45%/5.55%higher accuracy on ResNet32/ResNet18 on Cifar10/ImageNetas compared to Adam; and (iv) achieves 0.032% better score than Adagrad for DLRM on the Criteo Ad Kaggle dataset. The cost per iteration of ADAHESSIANis comparable to first-order methods, and ADAHESSIAN exhibits improved robustness towards variations in hyperparameter values. The code for ADAHESSIAN is open-sourced and publicly-available [1].

Authors

Z. Yao

2 papers

A. Gholami

3 papers

Sheng Shen

4 papers

TL;DR

Abstract

Authors

References84 items

Inexact Nonconvex Newton-Type Methods

BEAR: Sketching BFGS Algorithm for Ultra-High Dimensional Feature Selection in Sublinear Memory

Why are Adaptive Methods Good for Attention Models?

SqueezeBERT: What can computer vision teach NLP about efficient neural networks?

PyHessian: Neural Networks Through the Lens of the Hessian

PyTorch: An Imperative Style, High-Performance Deep Learning Library

HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

Fast Evaluation and Approximation of the Gauss-Newton Hessian Matrix for the Multilayer Perceptron

Why ADAM Beats SGD for Attention Models

On the Variance of the Adaptive Learning Rate and Beyond

A Tensorized Transformer for Language Modeling

Learning Deep Transformer Models for Machine Translation

Deep Learning Recommendation Model for Personalization and Recommendation Systems

HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

OverSketched Newton: Fast Convex Optimization for Serverless Systems

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Trust Region Based Adversarial Attack on Neural Networks

Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

Scaling Neural Machine Translation

How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift)

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Adafactor: Adaptive Learning Rates with Sublinear Memory Cost

Shampoo: Preconditioned Stochastic Tensor Optimization

Hessian-based Analysis of Large Batch Training and Robustness to Adversaries

Inexact Non-Convex Newton-Type Methods

A Progressive Batching L-BFGS Method for Machine Learning

Decoupled Weight Decay Regularization

Automatic differentiation in PyTorch

GIANT: Globally Improved Approximate Newton Method for Distributed Optimization

A Generic Approach for Escaping Saddle points

Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study

Newton-type methods for non-convex optimization under inexact Hessian information

Deep & Cross Network for Ad Click Predictions

Attention is All you Need

How to Escape Saddle Points Efficiently

The Power of Normalization: Faster Evasion of Saddle Points

Entropy-SGD: biasing gradient descent into wide valleys

Finding approximate local minima faster than gradient descent

Finding Approximate Local Minima for Nonconvex Optimization in Linear Time

Exact and Inexact Subsampled Newton Methods for Optimization

Pointer Sentinel Mixture Models

An overview of gradient descent optimization algorithms

SGDR: Stochastic Gradient Descent with Warm Restarts

Sub-sampled Newton Methods with Non-uniform Sampling

Optimization Methods for Large-Scale Machine Learning

SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size

Second Order Stochastic Optimization in Linear Time

Sub-Sampled Newton Methods I: Globally Convergent Algorithms

Deep Residual Learning for Image Recognition

Layer-Specific Adaptive Learning Rates for Deep Networks

Optimizing Neural Networks with Kronecker-factored Approximate Curvature

Escaping From Saddle Points - Online Stochastic Gradient for Tensor Decomposition

Adam: A Method for Stochastic Optimization

Deep learning with Elastic Averaging SGD

ADADELTA: An Adaptive Learning Rate Method

No more pesky learning rates

Convergence of adaptive BEM for some mixed boundary value problem

Empirical Evaluation and Combination of Advanced Language Modeling Techniques

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

Adaptive Bound Optimization for Online Convex Optimization

Convex optimization

ImageNet: A large-scale hierarchical image database

An estimator for the diagonal of a matrix

Bleu: a Method for Automatic Evaluation of Machine Translation

A Limited Memory Algorithm for Bound Constrained Optimization

Second Order Derivatives for Network Pruning: Optimal Brain Surgeon

INEXACT NEWTON METHODS

A Stochastic Approximation Method

Sequence Modeling

https://github.com/amirgholami/ADAHESSIAN.git. Github Online System

TRUST REGION METHODS

Conference on Learning Representations

Lecture 6.5rmsprop: Divide the gradient by a running average of its recent magnitude

Learning Multiple Layers of Features from Tiny Images