Simplified State Space Layers for Sequence Modeling (2022-08-09T00:00:00.000000Z)

TL;DR

A state space layer that can leverage efficient and widely implemented parallel scans, allowing S5 to match the computational efficiency of S4, while also achieving state-of-the-art performance on several long-range sequence modeling tasks.

Abstract

Models using structured state space sequence (S4) layers have achieved state-of-the-art performance on long-range sequence modeling tasks. An S4 layer combines linear state space models (SSMs), the HiPPO framework, and deep learning to achieve high performance. We build on the design of the S4 layer and introduce a new state space layer, the S5 layer. Whereas an S4 layer uses many independent single-input, single-output SSMs, the S5 layer uses one multi-input, multi-output SSM. We establish a connection between S5 and S4, and use this to develop the initialization and parameterization used by the S5 model. The result is a state space layer that can leverage efficient and widely implemented parallel scans, allowing S5 to match the computational efficiency of S4, while also achieving state-of-the-art performance on several long-range sequence modeling tasks. S5 averages 87.4% on the long range arena benchmark, and 98.5% on the most difficult Path-X task.

Authors

Jimmy Smith

1 papers

Andrew Warrington

1 papers

Scott W. Linderman

1 papers

TL;DR

Abstract

Authors

References82 items

Liquid Structural State-Space Models

Mega: Moving Average Equipped Gated Attention

How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections

On the Parameterization and Initialization of Diagonal State Space Models

Towards a General Purpose CNN for Long Range Dependencies in ND

Long Movie Clip Classification with State-Space Video Models

Diagonal State Spaces are as Effective as Structured State Spaces

It's Raw! Audio Generation with State-Space Models

Modeling Irregular Time Series with Continuous Recurrent Units

Efficiently Modeling Long Sequences with Structured State Spaces

Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes

H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences

Luna: Linear Unified Nested Attention

FNet: Mixing Tokens with Fourier Transforms

UnICORNN: A recurrent model for learning very long time dependencies

Parallelizing Legendre Memory Unit Training

Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention

CKConv: Continuous Kernel Convolution For Sequential Data

Long Range Arena: A Benchmark for Efficient Transformers

Rethinking Attention with Performers

Neural Rough Differential Equations for Long Time Series

HiPPO: Recurrent Memory with Optimal Polynomial Projections

Big Bird: Transformers for Longer Sequences

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Lipschitz Recurrent Neural Networks

Multi-Time Attention Networks for Irregularly Sampled Time Series

Linformer: Self-Attention with Linear Complexity

GMAT: Global Memory Augmentation for Transformers

Longformer: The Long-Document Transformer

Reformer: The Efficient Transformer

Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks

Temporal Parallelization of Bayesian Smoothers

GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series

Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces

AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks

Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group

Weighted Sigmoid Gate Unit for an Activation Function of Deep Neural Network

Trellis Networks for Sequence Modeling

Neural Ordinary Differential Equations

Learning long-range spatial dependencies with horizontal gated-recurrent units

Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

ListOps: A Diagnostic Dataset for Latent Tree Learning

Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

Adversarial Audio Synthesis

Learning Longer-term Dependencies in RNNs with Auxiliary Losses

Decoupled Weight Decay Regularization

Dilated Recurrent Neural Networks

Parallelizing Linear Recurrent Neural Nets Over Sequence Length

Simple Recurrent Units for Highly Parallelizable Recurrence

Attention is All you Need

Language Modeling with Gated Convolutional Networks

Quasi-Recurrent Neural Networks

WaveNet: A Generative Model for Raw Audio

Gaussian Error Linear Units (GELUs)

Unitary Evolution Recurrent Neural Networks

Fast approximate computations with Cauchy matrices and polynomials

Adam: A Method for Stochastic Optimization

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

On the difficulty of training recurrent neural networks

Learning Word Vectors for Sentiment Analysis

The ACL anthology network corpus

Long Short-Term Memory

On Parallel Prefix Computation

We sample these parameters in line with S4 and sample each element of log ∆ ∈ R from a uniform distribution on the interval

The Annotated S4

2022) shows the following relationship for scalar input signals as N →∞

window size of 2× 2 and a stride of 2. The dense layer has thirty hidden units. The linear readout layer outputs H = 30 features. This is chosen to match the encoding size

PENDULUM REGRESSION We use the same encoder-decoder architecture

The S4 and S4D layers also use the same S4 SSM latent size as reported

In-depth Benchmarking of Deep Neural Network Architectures for ECG Diagnosis

Linear Dynamical Systems as a Core Computational Primitive

Improving the Gating Mechanism of Recurrent Neural Networks

Latent Ordinary Differential Equations for Irregularly-Sampled Time Series

Pathfinder: Based off of the Pathfinder challenge