Deep Learning using Rectified Linear Units (ReLU)
Published in arXiv.org (2018)
TL
TL;DR
The use of rectified linear units (ReLU) as the classification function in a deep neural network (DNN) is introduced by taking the activation of the penultimate layer of a neural network, then multiplying it by weight parameters $\theta$ to get the raw scores.
Abstract
We introduce the use of rectified linear units (ReLU) as the classification function in a deep neural network (DNN). Conventionally, ReLU is used as an activation function in DNNs, with Softmax function as their classification function. However, there have been several studies on using a classification function other than Softmax, and this study is an addition to those. We accomplish this by taking the activation of the penultimate layer $h_{n - 1}$ in a neural network, then multiply it by weight parameters $\theta$ to get the raw scores $o_{i}$. Afterwards, we threshold the raw scores $o_{i}$ by $0$, i.e. $f(o) = \max(0, o_{i})$, where $f(o)$ is the ReLU function. We provide class predictions $\hat{y}$ through argmax function, i.e. argmax $f(x)$.
References17 items
1
A Neural Network Architecture Combining Gated Recurrent Unit (GRU) and Support Vector Machine (SVM) for Intrusion Detection in Network Traffic Data
2
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
3
Parametric Exponential Linear Unit for Deep Convolutional Neural Networks
4
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
5