Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2020-12-14T00:00:00.000000Z)

TL;DR

An efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: a ProbSparse self-attention mechanism, which achieves O(L log L) in time complexity and memory usage, and has comparable performance on sequences' dependency alignment.

Abstract

Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a ProbSparse self-attention mechanism, which achieves O(L log L) in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.

Authors

Hui Xiong

2 papers

Haoyi Zhou

1 papers

Shanghang Zhang

3 papers

TL;DR

Abstract

Authors

References57 items

Linformer: Self-Attention with Linear Complexity

Language Models are Few-Shot Learners

Longformer: The Long-Document Transformer

Reformer: The Efficient Transformer

Seq-U-Net: A One-Dimensional Causal U-Net for Efficient Sequence Modelling

Compressive Transformers for Long-Range Sequence Modelling

Blockwise Self-Attention for Long Document Understanding

Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel

Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation

Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer

Adaptively Truncating Backpropagation Through Time to Control Gradient Bias

Generating Long Sequences with Sparse Transformers

DSTP-RNN: a dual-stage two-phase attention-based recurrent neural networks for long-term and multivariate time series prediction

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

A Memory-Network Based Solution for Multivariate Time-Series Forecasting

ARMDN: Associative and Recurrent Mixture Density Networks for eRetail Demand Forecasting

Learning Longer-term Dependencies in RNNs with Auxiliary Losses

Convolutional Sequence Modeling Revisited

Forecasting at Scale

A Multi-Horizon Quantile Recurrent Forecaster

Attend and Diagnose: Clinical Time Series Analysis using Attention Models

Long-term Forecasting using Tensor-Train RNNs

Long-term Forecasting using Higher Order Tensor RNNs

Dilated Convolutions for Modeling Long-Distance Genomic Dependencies

Approximate Bayesian Inference in Linear State Space Models for Intermittent Demand Forecasting at Scale

Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting

Attention is All you Need

Dilated Residual Networks

DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks

A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction

Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks

Bayesian Intermittent Demand Forecasting for Large Inventories

Recurrent Highway Networks

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Effective Approaches to Attention-based Neural Machine Translation

On the Approximation of the Sum of Lognormals by a Log Skew Normal Distribution

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Sequence to Sequence Learning with Neural Networks

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Neural Machine Translation by Jointly Learning to Align and Translate

FUNNEL: automatic mining of spatially coevolving epidemics

Stock Price Prediction Using the ARIMA Model

Time Series: Theory and Methods

The Sum and Difference of Two Lognormal Random Variables

Optimal multi-scale patterns in time series streams

Change of Support of Transformations: Conservation of Lognormality Revisited

Broad distribution effects in sums of lognormal random variables

StatStream: Statistical Monitoring of Thousands of Data Streams in Real Time

Long Short-Term Memory

Bidirectional recurrent neural networks

The meaning and measurement of size hierarchies in plant populations

Time Series Analysis: Forecasting and Control

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Log-sum-exp neural networks and posynomial models for convex and log-log-convex data

An Extended Limit Theorem for Correlated Lognormal Sums

SUMS OF LOGNORMALS

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names