Transformers ar… (2020-06-29T00:00:00.000000Z)

TL;DR

This work expresses the self-attention as a linear dot-product of kernel feature maps and makes use of the associativity property of matrix products to reduce the complexity from O(N) to N, where N is the sequence length.

Abstract

Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.

Authors

Angelos Katharopoulos

2 papers

Franccois Fleuret

2 papers

Apoorv Vyas

2 papers

References43 items

End-to-End Object Detection with Transformers

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Reformer: The Efficient Transformer

PyTorch: An Imperative Style, High-Performance Deep Learning Library

On the Relationship between Self-Attention and Convolutional Layers

TL;DR

Abstract

Authors

Angelos Katharopoulos

2 papers

Franccois Fleuret

2 papers

Apoorv Vyas

2 papers

TL;DR

Abstract

Authors

References43 items

End-to-End Object Detection with Transformers

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Reformer: The Efficient Transformer

PyTorch: An Imperative Style, High-Performance Deep Learning Library

On the Relationship between Self-Attention and Convolutional Layers

TL;DR

Abstract

Authors

References43 items

End-to-End Object Detection with Transformers

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Reformer: The Efficient Transformer

PyTorch: An Imperative Style, High-Performance Deep Learning Library

On the Relationship between Self-Attention and Convolutional Layers

Q8BERT: Quantized 8Bit BERT

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Transformer Dissection: An Unified Understanding for Transformer’s Attention via the Lens of Kernel

On the Variance of the Adaptive Learning Rate and Beyond

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Sampled Softmax with Random Fourier Features

Large Memory Layers with Product Keys

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Stand-Alone Self-Attention in Vision Models

Adaptive Attention Span in Transformers

MASS: Masked Sequence to Sequence Pre-training for Language Generation

Are Sixteen Heads Really Better than One?

Generating Long Sequences with Sparse Transformers

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Efficient Attention: Attention with Linear Complexities

Universal Transformers

Self-Attentional Acoustic Models

Adaptive Sampled Softmax with Kernel Based Sampling

Attention is All you Need

PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Adam: A Method for Stochastic Optimization

Sequence to Sequence Learning with Neural Networks

Neural Machine Translation by Jointly Learning to Align and Translate

A Scalable Hierarchical Distributed Language Model

Classes for fast maximum entropy training

Long Short-Term Memory

The Design for the Wall Street Journal-based CSR Corpus

Pre-training text encoders as discriminators rather than generators

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Improving Language Understanding by Generative Pre-Training

Mnist handwritten digit database

Learning Multiple Layers of Features from Tiny Images

Hierarchical Probabilistic Neural Network Language Model

Connectionist Temporal Classiﬁcation: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

Transformers are RNNs

Connectionist Temporal Classiﬁcation: Labelling Unsegmented Sequences with Recurrent Neural Networks

Field of Study

Venue Information

Name

Type

URL

Alternate Names