Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (2023-05-26T00:00:00.000000Z)

TL;DR

This study investigates how weight decay affects the update behavior of individual neurons in deep neural networks through a combination of applied analysis and experimentation, offering a new simple perspective on training that elucidates the efficacy of widely used but poorly understood methods in deep learning.

Abstract

This study investigates how weight decay affects the update behavior of individual neurons in deep neural networks through a combination of applied analysis and experimentation. Weight decay can cause the expected magnitude and angular updates of a neuron's weight vector to converge to a steady state we call rotational equilibrium. These states can be highly homogeneous, effectively balancing the average rotation -- a proxy for the effective learning rate -- across different layers and neurons. Our work analyzes these dynamics across optimizers like Adam, Lion, and SGD with momentum, offering a new simple perspective on training that elucidates the efficacy of widely used but poorly understood methods in deep learning. We demonstrate how balanced rotation plays a key role in the effectiveness of normalization like Weight Standardization, as well as that of AdamW over Adam with L2-regularization. Finally, we show that explicitly controlling the rotation provides the benefits of weight decay while substantially reducing the need for learning rate warmup.

Authors

Martin Jaggi

9 papers

Atli Kosson

1 papers

Bettina Messmer

1 papers

TL;DR

Abstract

Authors

References66 items

High Performance Large Scale Image Recognition Without Normalization

Analyzing and Improving the Training Dynamics of Diffusion Models

Why Do We Need Weight Decay in Modern Deep Learning?

When and Why Momentum Accelerates SGD: An Empirical Study

Ghost Noise for Regularizing Deep Neural Networks

Symbolic Discovery of Optimization Algorithms

Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes

On the SDEs and Scaling Rules for Adaptive Gradient Algorithms

Training Compute-Optimal Large Language Models

Robust Training of Neural Networks using Scale Invariant Architectures

FixNorm: Dissecting Weight Decay for Training Deep Neural Networks

On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)

Learning by Turning: Neural Architecture Aware Optimisation

High-Performance Large-Scale Image Recognition Without Normalization

Characterizing signal propagation to close the performance gap in unnormalized ResNets

Training data-efficient image transformers & distillation through attention

On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective

Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate

A spherical analysis of Adam with Batch Normalization

AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights

PyTorch: An Imperative Style, High-Performance Deep Learning Library

An Exponential Learning Rate Schedule for Deep Learning

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Online Normalization for Training Neural Networks

Large Batch Optimization for Deep Learning: Training BERT in 76 minutes

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Weight Standardization

An Empirical Model of Large-Batch Training

Measuring the Effects of Data Parallelism on Neural Network Training

Three Mechanisms of Weight Decay Regularization

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

Group Normalization

Norm matters: efficient and accurate normalization schemes in deep networks

Decoupled Weight Decay Regularization

Projection Based Weight Normalization for Deep Neural Networks

Centered Weight Normalization in Accelerating Training of Deep Neural Networks

Large Batch Training of Convolutional Networks

L2 Regularization versus Batch and Weight Normalization

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Pointer Sentinel Mixture Models

Layer Normalization

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

Deep Residual Learning for Image Recognition

Path-SGD: Path-Normalized Optimization in Deep Neural Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Fast Mixing of Stochastic Gradient Descent with Normalization and Weight Decay

2022b). A.3 PROJECTED OPTIMIZATION Some existing works are based on projections onto a sphere but do not scale the update to be proportional to the weight magnitude

Rotational Optimizer Variants keep the relative learning rate at a fixed level and do not suffer from the effect

Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay

How Weight Decay Balances Learning Across Neural Networks

Rotational Equilibrium: How Weight Decay Balances

Language Models are Unsupervised Multitask Learners

Pytorch image models

2019) also known as Centered Weight Normalization (Huang et al., 2017), where each filter is scale and shift invariant

Deep Learning

D-1 We pre-process the data by normalizing it with mean (0 . 49140 . 48220 . 4465) and std (0 . 20230 . 19940 . 2010)

D-2 We use the standard data augmentation from He et al.

Report on the 11th IWSLT evaluation campaign

Learning Multiple Layers of Features from Tiny Images

Some methods of speeding up the convergence of iteration methods

We focus on the case where the weights are scale-invariant, defining ˜ g = ∥ ω ∥ g as the gradient corresponding to a unit weight norm based on the inverse proportionality from Equation 6

we list additional details referenced in the

D-6 For this experiment we use the nanoGPT library (Karpathy, 2023)

Field of Study

Venue Information

Name

Type

URL

Alternate Names