AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
Published in
International Conference on Learning Representa...(2020)External Links:
TL
TL;DR
This paper proposes a simple and effective remedy, SGDP and AdamP: get rid of the radial component, or the norm-increasing direction, at each optimizer step, which alters the effective step sizes without changing the effective update directions, thus enjoying the original convergence properties of GD optimizers.