Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019-04-06T00:00:00.000000Z)

TL;DR

This work proposes Iterative Normalization (IterNorm), which employs Newton’s iterations for much more efficient whitening, while simultaneously avoiding the eigen-decomposition, and exclusively introduces Stochastic Normalization Disturbance (SND), which measures the inherent stochastic uncertainty of samples when applied to normalization operations.

Abstract

Batch Normalization (BN) is ubiquitously employed for accelerating neural network training and improving the generalization capability by performing standardization within mini-batches. Decorrelated Batch Normalization (DBN) further boosts the above effectiveness by whitening. However, DBN relies heavily on either a large batch size, or eigen-decomposition that suffers from poor efficiency on GPUs. We propose Iterative Normalization (IterNorm), which employs Newton’s iterations for much more efficient whitening, while simultaneously avoiding the eigen-decomposition. Furthermore, we develop a comprehensive study to show IterNorm has better trade-off between optimization and generalization, with theoretical and experimental support. To this end, we exclusively introduce Stochastic Normalization Disturbance (SND), which measures the inherent stochastic uncertainty of samples when applied to normalization operations. With the support of SND, we provide natural explanations to several phenomena from the perspective of optimization, e.g., why group-wise whitening of DBN generally outperforms full-whitening and why the accuracy of BN degenerates with reduced batch sizes. We demonstrate the consistently improved performance of IterNorm with extensive experiments on CIFAR-10 and ImageNet over BN and DBN.

Authors

Ling Shao

32 papers

Yi Zhou

5 papers

Fan Zhu

3 papers

TL;DR

Abstract

Authors

References53 items

Differentiable Learning-to-Normalize via Switchable Normalization

Understanding Batch Normalization

How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift)

Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization

Towards a Theoretical Understanding of Batch Normalization

Decorrelated Batch Normalization

Normalization of Neural Networks using Analytic Variance Propagation

Group Normalization

Norm matters: efficient and accurate normalization schemes in deep networks

$L1$ -Norm Batch Normalization for Efficient Training of Deep Neural Networks

Bayesian Uncertainty Estimation for Batch Normalized Deep Networks

Uncertainty Estimation via Stochastic Batch Normalization

Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization

Centered Weight Normalization in Accelerating Training of Deep Neural Networks

Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

Learning Deep Architectures via Generalized Whitened Neural Networks

Improved Bilinear Pooling with CNNs

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Self-Normalizing Neural Networks

Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models

Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint

Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes

Regularizing CNNs with Locally Constrained Decorrelations

Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning

Densely Connected Convolutional Networks

Layer Normalization

Wide Residual Networks

Recurrent Batch Normalization

Identity Mappings in Deep Residual Networks

Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Reducing Overfitting in Deep Networks by Decorrelating Representations

Batch normalized recurrent neural networks

Natural Neural Networks

Path-SGD: Path-Normalized Optimization in Deep Neural Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

cuDNN: Efficient Primitives for Deep Learning

Very Deep Convolutional Networks for Large-Scale Image Recognition

Mean-normalized stochastic gradient for large-scale deep learning

Deep Boltzmann Machines and the Centering Trick

Deep Learning Made Easier by Linear Transformations in Perceptrons

ImageNet: A large-scale hierarchical image database

Algorithms for the matrix pth root

Accelerated Gradient Descent by Factor-Centering Decomposition

A Simple Weight Decay Can Improve Generalization

Kalman Normalization: Normalizing Internal Representations Across Network Layers

Dropout: a simple way to prevent neural networks from overfitting

Torch7: A Matlab-like Environment for Machine Learning

Learning Multiple Layers of Features from Tiny Images

Effiicient BackProp

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names