iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (2023-10-10T00:00:00.000000Z)

TL;DR

The iTransformer model achieves state-of-the-art on challenging real-world datasets, which further empowers the Transformer family with promoted performance, generalization ability across different variates, and better utilization of arbitrary lookback windows, making it a nice alternative as the fundamental backbone of time series forecasting.

Abstract

The recent boom of linear forecasting models questions the ongoing passion for architectural modifications of Transformer-based forecasters. These forecasters leverage Transformers to model the global dependencies over temporal tokens of time series, with each token formed by multiple variates of the same timestamp. However, Transformers are challenged in forecasting series with larger lookback windows due to performance degradation and computation explosion. Besides, the embedding for each temporal token fuses multiple variates that represent potential delayed events and distinct physical measurements, which may fail in learning variate-centric representations and result in meaningless attention maps. In this work, we reflect on the competent duties of Transformer components and repurpose the Transformer architecture without any modification to the basic components. We propose iTransformer that simply applies the attention and feed-forward network on the inverted dimensions. Specifically, the time points of individual series are embedded into variate tokens which are utilized by the attention mechanism to capture multivariate correlations; meanwhile, the feed-forward network is applied for each variate token to learn nonlinear representations. The iTransformer model achieves state-of-the-art on challenging real-world datasets, which further empowers the Transformer family with promoted performance, generalization ability across different variates, and better utilization of arbitrary lookback windows, making it a nice alternative as the fundamental backbone of time series forecasting. Code is available at this repository: https://github.com/thuml/iTransformer.

Authors

References38 items

TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting

Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors

Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping

Long-term Forecasting with TiDE: Time-series Dense Encoder

The Capacity and Robustness Trade-Off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecasting

SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling

A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Are Transformers Effective for Time Series Forecasting?

Flowformer: Linearizing Transformers with Conservation Flows

FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting

Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction

MLP-Mixer: An all-MLP Architecture for Vision

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Language Models are Few-Shot Learners

Customer

Scaling Laws for Neural Language Models

Reformer: The Efficient Transformer

PyTorch: An Imperative Style, High-Performance Deep Learning Library

N-BEATS: Neural basis expansion analysis for interpretable time series forecasting

Similarity of Neural Network Representations Revisited

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

Attention is All you Need

DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks

Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks

LSTM network: a deep learning approach for short-term traffic forecast

Layer Normalization

Adam: A Method for Stochastic Optimization

Cryptosporidium Parvum Genome Project

Approximation capabilities of multilayer feedforward networks

Some Recent Advances in Forecasting and Control

Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting

Non-stationary Transformers: Rethinking the Stationarity in Time Series Forecasting

Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift

Deep State Space Models for Time Series Forecasting