Resurrecting Recurrent Neural Networks for Long Sequences (2023-03-11T00:00:00.000000Z)

TL;DR

This paper shows that careful design of deep RNNs using standard signal propagation arguments can recover the impressive performance of deep SSMs on long-range reasoning tasks, whileAlso introducing an RNN block called the Linear Recurrent Unit that matches both their performance on the Long Range Arena benchmark and their computational efficiency.

Abstract

Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train. Deep state-space models (SSMs) have recently been shown to perform remarkably well on long sequence modeling tasks, and have the added benefits of fast parallelizable training and RNN-like fast inference. However, while SSMs are superficially similar to RNNs, there are important differences that make it unclear where their performance boost over RNNs comes from. In this paper, we show that careful design of deep RNNs using standard signal propagation arguments can recover the impressive performance of deep SSMs on long-range reasoning tasks, while also matching their training speed. To achieve this, we analyze and ablate a series of changes to standard RNNs including linearizing and diagonalizing the recurrence, using better parameterizations and initializations, and ensuring proper normalization of the forward pass. Our results provide new insights on the origins of the impressive performance of deep SSMs, while also introducing an RNN block called the Linear Recurrent Unit that matches both their performance on the Long Range Arena benchmark and their computational efficiency.

Authors

Caglar Gulcehre

5 papers

Razvan Pascanu

15 papers

Albert Gu

4 papers

TL;DR

Abstract

Authors

References116 items

Hungry Hungry Hippos: Towards Language Modeling with State Space Models

Novel machine learning approaches revolutionize protein knowledge

Simplifying and Understanding State Space Models with Diagonal Linear RNNs

What Makes Convolutional Models Great on Long Sequence Modeling?

Liquid Structural State-Space Models

Mega: Moving Average Equipped Gated Attention

Simplified State Space Layers for Sequence Modeling

Long Range Language Modeling via Gated State Spaces

How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections

On the Parameterization and Initialization of Diagonal State Space Models

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Learning Dynamical Systems via Koopman Operator Regression in Reproducing Kernel Hilbert Spaces

FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting

Long Movie Clip Classification with State-Space Video Models

Diagonal State Spaces are as Effective as Structured State Spaces

It's Raw! Audio Generation with State-Space Models

Efficiently Modeling Long Sequences with Structured State Spaces

Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

Highly accurate protein structure prediction with AlphaFold

FNet: Mixing Tokens with Fourier Transforms

Recognizing and Verifying Mathematical Equations using Multiplicative Differential Neural Units

Long Range Arena: A Benchmark for Efficient Transformers

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

HiPPO: Recurrent Memory with Optimal Polynomial Projections

Lipschitz Recurrent Neural Networks

Liquid Time-constant Networks

A provably stable neural network Turing Machine

Language Models are Few-Shot Learners

Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

Koopman Model Predictive Control of Nonlinear Dynamical Systems

Target Propagation in Recurrent Neural Networks

MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models

Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks

Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group

Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representations

Sparse Attentive Backtracking: Temporal CreditAssignment Through Reminding

Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

State Space LSTM Models with Particle MCMC Inference

Decoupled Weight Decay Regularization

Sparse Attentive Backtracking: Long-Range Credit Assignment in Recurrent Networks

Parallelizing Linear Recurrent Neural Nets Over Sequence Length

Orthogonal Recurrent Neural Networks with Scaled Cayley Transform

Data-driven discovery of Koopman eigenfunctions for control

Attention is All you Need

On Convergence of Extended Dynamic Mode Decomposition to the Koopman Operator

Unbiased Online Recurrent Optimization

Language Modeling with Gated Convolutional Networks

Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs

Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections

Koopman operator based observer synthesis for control-affine nonlinear systems

Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems

Neural Machine Translation in Linear Time

Full-Capacity Unitary Recurrent Neural Networks

Gradient Descent Learns Linear Dynamical Systems

WaveNet: A Generative Model for Raw Audio

Layer Normalization

Learning Unitary Operators with Help From u(n)

A Random Matrix Approach to Echo-State Neural Networks

Generalizing Koopman Theory to Allow for Inputs and Control

Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

Unitary Evolution Recurrent Neural Networks

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

A Data–Driven Approximation of the Koopman Operator: Extending Dynamic Mode Decomposition

Global Stability Analysis Using the Eigenfunctions of the Koopman Operator

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

On the difficulty of training recurrent neural networks

Understanding the difficulty of training deep feedforward neural networks

Jordan Canonical Form: Theory and Practice

Reservoir computing approaches to recurrent neural network training

Dynamic mode decomposition of numerical and experimental data

Eigenvalue spectra of random matrices for neural networks.