Adaptive hybrid activation function for deep neural networks (2022-04-25T00:00:00.000000Z)

TL;DR

The adaptive hybrid activation function (AHAF) is proposed that combines the properties of the rectifier units and the squashing functions and provides high performance for the neural network training process.

Abstract

The adaptive hybrid activation function (AHAF) is proposed that combines the properties of the rectifier units and the squashing functions. The proposed function can be used as a drop-in replacement for ReLU, SiL and Swish activations for deep neural networks and can evolve to one of such functions during the training. The effectiveness of the function was evaluated on the image classification task using the Fashion-MNIST and CIFAR-10 datasets. The evaluation shows that the neural networks with AHAF activations achieve better classification accuracy comparing to their base implementations that use ReLU and SiL. A double-stage parameter tuning process for training the neural networks with AHAF is proposed. The proposed approach is sufficiently simple from the implementation standpoint and provides high performance for the neural network training process.

Authors

Y. Bodyanskiy

1 papers

Serhii Kostiuk

2 papers

TL;DR

Abstract

Authors

References33 items

Universal activation function for machine learning

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Weighted Sigmoid Gate Unit for an Activation Function of Deep Neural Network

Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks

Learning Combinations of Activation Functions

Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning

Deep Learning Neural Networks - Design and Case Studies

Deep Learning with S-Shaped Rectified Linear Activation Units

Deep Residual Learning for Image Recognition

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Adam: A Method for Stochastic Optimization

Learning Activation Functions to Improve Deep Neural Networks

Deep learning in neural networks: An overview

A new learning algorithm for a forecasting neuro-fuzzy network

Networks with trainable amplitude of activation functions

Long Short-Term Memory

A feedforward neural network with function shape autotuning

Approximation capabilities of multilayer feedforward networks

Approximation by superpositions of a sigmoidal function

Mish: A Self Regularized Non-Monotonic Activation Function

Formal Neuron Based on Adaptive Parametric Rectified Linear Activation Function and its Learning

Searching for Activation Functions

Deep Learning

Learning Multiple Layers of Features from Tiny Images

Gradient-based learning applied to document recognition

Neural networks for optimization and signal processing

Benefits of gain: speeded learning and minimal hidden layers in back-propagation networks

Keras

S . Hochreiter and J . Schmidhuber , “ Long Short - Term Memory ”

Neural Networks and Deep Learning: A Textbook, 1st ed

The study of neural network adaptive control systems

Deep Neural Networks in a Mathematical Framework, 1st ed

Field of Study

Journal Information

Name