Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015-11-23T00:00:00.000000Z)

TL;DR

The "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies and significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers.

Abstract

We introduce the "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information. Therefore, ELUs code the degree of presence of particular phenomena in the input, while they do not quantitatively model the degree of their absence. In experiments, ELUs lead not only to faster learning, but also to significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with batch normalization while batch normalization does not improve ELU networks. ELU networks are among the top 10 reported CIFAR-10 results and yield the best published result on CIFAR-100, without resorting to multi-view evaluation or model averaging. On ImageNet, ELU networks considerably speed up learning compared to a ReLU network with the same architecture, obtaining less than 10% classification error for a single crop, single model network.

References67 items

DeepTox: Toxicity Prediction using Deep Learning

Training Very Deep Networks

Scaling up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix

Natural Neural Networks

Empirical Evaluation of Rectified Activations in Convolutional Network

Toxicity Prediction using Deep Learning

Rectified Factor Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Striving for Simplicity: The All Convolutional Net

Fractional Max-Pooling

Deeply-Supervised Nets

Network In Network

On rectified linear units for speech processing

Improving deep neural networks for LVCSR using rectified linear units and dropout

Riemannian metrics for neural networks I: feedforward networks

Riemannian metrics for neural networks

Maxout Networks

Training Neural Networks with Stochastic Hessian-Free Optimization

Metric-Free Natural Gradient for Joint-Training of Boltzmann Machines

Revisiting Natural Gradient for Deep Networks

ImageNet classification with deep convolutional neural networks

Deep Boltzmann Machines and the Centering Trick

Deep Learning Made Easier by Linear Transformations in Perceptrons

Krylov Subspace Descent for Deep Learning

Enhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines

Learning Recurrent Neural Networks with Hessian-Free Optimization

Deep Sparse Rectifier Neural Networks

Deep learning via Hessian-free optimization

Rectified Linear Units Improve Restricted Boltzmann Machines

A fast natural Newton method

Stochastic search using the natural gradient

Natural Evolution Strategies

Topmoumoute Online Natural Gradient Algorithm

Incremental Natural Actor-Critic Algorithms

Iterative Scaled Trust-Region Learning in Krylov Subspaces via Pearlmutter's Implicit Sparse Hessian-Vector Multiply

Natural Actor-Critic

A Natural Policy Gradient

Adaptive natural gradient learning algorithms for various stochastic models

A Fast, Compact Approximation of the Exponential Function

Feature Extraction Through LOCOCODE

Complexity Issues in Natural Gradient Descent Method for Training Multilayer Perceptrons

The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions

Natural Gradient Works Efficiently in Learning

Long Short-Term Memory

Neural Learning in Structured Parameter Spaces - Natural Riemannian Gradient

A New Learning Algorithm for Blind Signal Separation

Statistical Theory of Learning Curves under Entropic Loss Criterion

Iterative weighted least squares algorithms for neural networks classifiers

Eigenvalues of covariance matrices: Application to neural-network learning.

Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS

Under review as a conference paper at ICLR 2019 MILE : A Multi-Level Framework for Scalable Graph Embedding

Deep Learning as an Opportunity in Virtual Screening

Learning Semantic Image Representations at a Large Scale

J. Natural evolution strategies. Journal of Machine Learning Research

Rectifier Nonlinearities Improve Neural Network Acoustic Models

Efficient BackProp

Improved Preconditioner for Hessian Free Optimization

Neural Networks: Tricks of the Trade

Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies

NATURAL GRADIENT LEARNING WITH A NONHOLONOMIC CONSTRAINT FOR BLIND DECONVOLUTION OF MULTIPLE CHANNELS

Natural Gradient Approach To Blind Separation Of Over- And Under-Complete Mixtures