A Transformer-based Framework for Multivariate Time Series Representation Learning

Published in

Knowledge Discovery and Data Mining(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

A novel framework for multivariate time series representation learning based on the transformer encoder architecture, which can offer substantial performance benefits over fully supervised learning on downstream tasks, both with but even without leveraging additional unlabeled data, i.e., by reusing the existing data samples.

Abstract

We present a novel framework for multivariate time series representation learning based on the transformer encoder architecture. The framework includes an unsupervised pre-training scheme, which can offer substantial performance benefits over fully supervised learning on downstream tasks, both with but even without leveraging additional unlabeled data, i.e., by reusing the existing data samples. Evaluating our framework on several public multivariate time series datasets from various domains and with diverse characteristics, we demonstrate that it performs significantly better than the best currently available methods for regression and classification, even for datasets which consist of only a few hundred training samples. Given the pronounced interest in unsupervised learning for nearly all domains in the sciences and in industry, these findings represent an important landmark, presenting the first unsupervised method shown to push the limits of state-of-the-art performance for multivariate time series regression and classification.

Authors

George Zerveas

1 papers

Srideepika Jayaraman

1 papers

Dhaval Patel

1 papers

References39 items

MiniRocket: A Very Fast (Almost) Deterministic Transform for Time Series Classification

Time series extrinsic regression

Monash University, UEA, UCR Time Series Regression Archive

Language Models are Few-Shot Learners

Longformer: The Long-Document Transformer

A Transformer-based Framework for Multivariate Time Series Representation Learning

Published in

Knowledge Discovery and Data Mining(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

George Zerveas

1 papers

Srideepika Jayaraman

1 papers

Dhaval Patel

1 papers

References39 items

MiniRocket: A Very Fast (Almost) Deterministic Transform for Time Series Classification

Time series extrinsic regression

Monash University, UEA, UCR Time Series Regression Archive

Language Models are Few-Shot Learners

Longformer: The Long-Document Transformer

A. Bhamidipaty

1 papers

Carsten Eickhoff

2 papers

PowerNorm: Rethinking Batch Normalization in Transformers

Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case

Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting

Learning representations of multivariate time series with missing data

ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Mixture-of-Experts Variational Autoencoder for clustering and generating from similarity-based representations

InceptionTime: Finding AlexNet for time series classification

On the Variance of the Adaptive Learning Rate and Beyond

Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

TS-CHIEF: a scalable and accurate forest algorithm for time series classification

GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series

CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation

Unsupervised Scalable Representation Learning for Multivariate Time Series

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Improving Clinical Predictions through Unsupervised Time Series Representation Learning

A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data

The UEA multivariate time series classification archive, 2018

Deep learning for time series classification: a review

Music Transformer: Generating Music with Long-Term Structure

Proximity Forest: an effective and scalable distance-based classifier for time series

Time Series Classification with HIVE-COTE

Unsupervised Learning of Semantic Audio Representations

TimeNet: Pre-trained deep recurrent neural network for time series classification

Attention is All you Need

Similarity Preserving Representation Learning for Time Series Analysis

The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances

Time Series Regression

On the difficulty of training recurrent neural networks

The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

SOM-VAE: Interpretable Discrete Representation Learning on Time Series

Time Series Regression

Field of Study

Computer Science

Journal Information

Name

Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

Venue Information

Name

Knowledge Discovery and Data Mining

Type

conference

URL

http://www.acm.org/sigkdd/

Alternate Names

KDD
Knowl Discov Data Min