A Transformer-based Framework for Multivariate Time Series Representation Learning (2020-10-06T00:00:00.000000Z)

TL;DR

A novel framework for multivariate time series representation learning based on the transformer encoder architecture, which can offer substantial performance benefits over fully supervised learning on downstream tasks, both with but even without leveraging additional unlabeled data, i.e., by reusing the existing data samples.

Abstract

We present a novel framework for multivariate time series representation learning based on the transformer encoder architecture. The framework includes an unsupervised pre-training scheme, which can offer substantial performance benefits over fully supervised learning on downstream tasks, both with but even without leveraging additional unlabeled data, i.e., by reusing the existing data samples. Evaluating our framework on several public multivariate time series datasets from various domains and with diverse characteristics, we demonstrate that it performs significantly better than the best currently available methods for regression and classification, even for datasets which consist of only a few hundred training samples. Given the pronounced interest in unsupervised learning for nearly all domains in the sciences and in industry, these findings represent an important landmark, presenting the first unsupervised method shown to push the limits of state-of-the-art performance for multivariate time series regression and classification.

Authors

George Zerveas

1 papers

Srideepika Jayaraman

1 papers

Dhaval Patel

1 papers

TL;DR

Abstract

Authors

References39 items

MiniRocket: A Very Fast (Almost) Deterministic Transform for Time Series Classification

Time series extrinsic regression

Monash University, UEA, UCR Time Series Regression Archive

Language Models are Few-Shot Learners

Longformer: The Long-Document Transformer

PowerNorm: Rethinking Batch Normalization in Transformers

Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case

Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting

Learning representations of multivariate time series with missing data

ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Mixture-of-Experts Variational Autoencoder for clustering and generating from similarity-based representations

InceptionTime: Finding AlexNet for time series classification

On the Variance of the Adaptive Learning Rate and Beyond

Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

TS-CHIEF: a scalable and accurate forest algorithm for time series classification

GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series

CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation

Unsupervised Scalable Representation Learning for Multivariate Time Series

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Improving Clinical Predictions through Unsupervised Time Series Representation Learning

A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data

The UEA multivariate time series classification archive, 2018

Deep learning for time series classification: a review

Music Transformer: Generating Music with Long-Term Structure

Proximity Forest: an effective and scalable distance-based classifier for time series

Time Series Classification with HIVE-COTE

Unsupervised Learning of Semantic Audio Representations

TimeNet: Pre-trained deep recurrent neural network for time series classification

Attention is All you Need

Similarity Preserving Representation Learning for Time Series Analysis

The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances

Time Series Regression

On the difficulty of training recurrent neural networks

The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions

University, UEA,

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

SOM-VAE: Interpretable Discrete Representation Learning on Time Series

Time Series Regression

Field of Study

Journal Information

Name

Venue Information

Name

Type

URL

Alternate Names