Activation functions in deep learning: A comprehensive survey and benchmark (2021-09-29T00:00:00.000000Z)

TL;DR

A comprehensive overview and survey is presented for AFs in neural networks for deep learning, covering different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based.

Authors

S. Dubey

4 papers

S. Singh

2 papers

B. Chaudhuri

2 papers

TL;DR

Authors

References159 items

A weight initialization method based on neural network with asymmetric activation function

Deep Learning for Weakly-Supervised Object Detection and Localization: A Survey

Review the state-of-the-art technologies of semantic segmentation based on deep learning

GAN-based anomaly detection: A review

A Decade Survey of Content Based Image Retrieval Using Deep Learning

Skin disease diagnosis with deep learning: a review

Elastic exponential linear units for convolutional neural networks

Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks

A survey on modern trainable activation functions

YOLOv4: Optimal Speed and Accuracy of Object Detection

Rational neural networks

Dynamic ReLU

The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition

Soft-Root-Sign Activation Function

Wide Hidden Expansion Layer for Deep Convolutional Neural Networks

Parametric Deformable Exponential Linear Units for deep neural networks

Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks

PCSGAN: Perceptual Cyclic-Synthesized Generative Adversarial Networks for Thermal and NIR to Visible Image Transformation

Learning Neural Activations

Symmetrical Gaussian Error Linear Units (SGELUs)

Time/Accuracy Tradeoffs for Learning a ReLU with respect to Gaussian Marginals

L*ReLU: Piece-wise Linear Activation Functions for Deep Fine-grained Visual Categorization

Improved Convolutional Neural Network Based on Fast Exponentially Linear Unit Activation Function

ReLTanh: An activation function with vanishing gradient resistance for SAE-based DNNs and its application to rotating machinery fault diagnosis

Fast Image Restoration With Multi-Bin Trainable Linear Units

An Evaluation of Parametric Activation Functions for Deep Learning

Improving Weight Initialization of ReLU and Output Layers

diffGrad: An Optimization Method for Convolutional Neural Networks

PowerNet: Efficient Representations of Polynomials and Smooth Functions by Deep Neural Networks with Rectified Power Units

Mish: A Self Regularized Non-Monotonic Neural Activation Function

Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks

Evaluation of maxout activations in deep learning across several big data domains

Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks

Rectified Exponential Units for Convolutional Neural Networks

Learning Activation Functions: A new paradigm of understanding Neural Networks

A Convolutional Neural Network Model Based on Improved Softplus Activation Function

Adaptive activation functions accelerate convergence in deep and physics-informed neural networks

Enhancing batch normalized convolutional networks using displaced rectifier linear units: A systematic comparative study

ProbAct: A Probabilistic Activation Function for Deep Neural Networks

Improving the Antinoise Ability of DNNs via a Bio-Inspired Noise Adaptive Activation Function Rand Softplus

Ensemble of Convolutional Neural Networks Trained with Different Activation Functions

The Optimized Deep Belief Networks With Improved Logistic Sigmoid Units and Their Application in Fault Diagnosis for Planetary Gearboxes of Wind Turbines

An Empirical Study on Generalizations of the ReLU Activation Function

Linearized sigmoidal activation: A novel activation function with tractable non-linear characteristics to boost representation capability

A simple and efficient architecture for trainable activation functions

Widely Linear Kernels for Complex-valued Kernel Activation Functions

Kafnets: Kernel-based non-parametric activation functions for neural networks

Overfitting remedy by sparsifying regularization on fully-connected layers of CNNs

Multikernel activation functions: formulation and a case study

Activation Adaptation in Neural Networks

Impact of Fully Connected Layers on Performance of Convolutional Neural Networks for Image Classification

Activation Functions for Generalized Learning Vector Quantization - A Performance Comparison

The Benefits of Over-parameterization at Initialization in Deep ReLU Networks

Piecewise Polynomial Activation Functions for Feedforward Neural Networks

A Performance Evaluation of Loss Functions for Deep Face Recognition

Why ReLU Networks Yield High-Confidence Predictions Far Away From the Training Data and How to Mitigate the Problem

Singular Values for ReLU Layers

Review of Adaptive Activation Function in Deep Neural Network

Activation Functions: Comparison of trends in Practice and Research for Deep Learning

A Dynamic ReLU on Neural Network

Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks

PLU: The Piecewise Linear Unit Activation Function

A Novel Parameterized Activation Function in Visual Geometry Group

The Quest for the Golden Activation Function

FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks

Improved Learning in Convolutional Neural Networks with Shifted Exponential Linear Units (ShELUs)

Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning

vReLU Activation Functions for Artificial Neural Networks

A novel softplus linear unit for deep convolutional neural networks

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks

Initialization of ReLUs for Dynamical Isometry

ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets

Symmetric Power Activation Functions for Deep Neural Networks

Singularities of Three-Layered Complex-Valued Neural Networks With Split Activation Function

Comparison of non-linear activation functions for deep neural networks on MNIST classification task

A comparison of deep networks with ReLU activation function and linear spline-type methods

Average biased ReLU based CNN descriptor for improved face retrieval