Gating Revisited: Deep Multi-Layer RNNs That can be Trained (2019-09-25T00:00:00.000000Z)

TL;DR

A new STAckable Recurrent cell (STAR) is proposed for recurrent neural networks (RNNs), which has fewer parameters than widely used LSTM and GRU while being more robust against vanishing or exploding gradients.

Abstract

We propose a new STAckable Recurrent cell (STAR) for recurrent neural networks (RNNs), which has fewer parameters than widely used LSTM [16] and GRU [10] while being more robust against vanishing or exploding gradients. Stacking recurrent units into deep architectures suffers from two major limitations: (i) many recurrent cells (e.g., LSTMs) are costly in terms of parameters and computation resources; and (ii) deep RNNs are prone to vanishing or exploding gradients during training. We investigate the training of multi-layer RNNs and examine the magnitude of the gradients as they propagate through the network in the ”vertical” direction. We show that, depending on the structure of the basic recurrent unit, the gradients are systematically attenuated or amplified. Based on our analysis we design a new type of gated cell that better preserves gradient magnitude. We validate our design on a large number of sequence modelling tasks and demonstrate that the proposed STAR cell allows to build and train deeper recurrent architectures, ultimately leading to improved performance while being computationally more efficient.

Authors

K. Schindler

19 papers

J. D. Wegner

6 papers

Mehmet Ozgur Turkoglu

3 papers

TL;DR

Abstract

Authors

References51 items

The Jester Dataset: A Large-Scale Video Dataset of Human Gestures

R-Transformer: Recurrent Neural Network Enhanced Transformer

BreizhCrops: A Satellite Time Series Dataset for Crop Type Identification

Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces

AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks

Wide neural networks of any depth evolve as linear models under gradient descent

Attention in Convolutional LSTM for Gesture Recognition

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks

The unreasonable effectiveness of the forget gate

Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

Can recurrent neural networks warp time?

Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders

Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks

Long Short-Term Memory Kalman Filters: Recurrent Neural Estimators for Pose Regularization

Temporal Vegetation Modelling Using Long Short-Term Memory Networks for Crop Identification from Medium-Resolution Multi-spectral Satellite Images

Attention is All you Need

Diagonal rnns in symbolic music modeling

On orthogonality and learning recurrent networks with long term dependencies

Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition

Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections

Full-Capacity Unitary Recurrent Neural Networks

Recurrent Highway Networks

VideoLSTM convolves, attends and flows for action recognition

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

Recurrent Batch Normalization

Architectural Complexity Measures of Recurrent Neural Networks

Recurrent Orthogonal Networks and Long-Memory Tasks

Unitary Evolution Recurrent Neural Networks

A Neural Conversational Model

Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

Highway Networks

A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Gated Feedback Recurrent Neural Networks

Adam: A Method for Stochastic Optimization

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Sequence to Sequence Learning with Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Speech recognition with deep recurrent neural networks

ImageNet classification with deep convolutional neural networks

Harmonising Chorales by Probabilistic Inference

Long Short-Term Memory

Learning long-term dependencies with gradient descent is difficult

Building a Large Annotated Corpus of English: The Penn Treebank

Exploring the Depths of Recurrent Neural Networks with Stochastic Residual Learning

Learning Deep Architectures for AI

A Discriminative Model for Polyphonic Piano Transcription

Gradient-based learning applied to document recognition

Untersuchungen zu dynamischen neuronalen Netzen

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names