FNet: Mixing Tokens with Fourier Transforms (2021-05-09T00:00:00.000000Z)

TL;DR

The FNet model is significantly faster: when compared to the “efficient Transformers” on the Long Range Arena benchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all sequence lengths on GPUs (and across relatively shorter lengths on TPUs).

Abstract

We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the self-attention sublayers with simple linear transformations that “mix” input tokens. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder with a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE benchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths, our FNet model is significantly faster: when compared to the “efficient Transformers” on the Long Range Arena benchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all sequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint and is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models outperform Transformer counterparts.

Authors

J. Ainslie

3 papers

J. Lee-Thorp

1 papers

Ilya Eckstein

1 papers

TL;DR

Abstract

Authors

References78 items

LongT5: Efficient Text-To-Text Transformer for Long Sequences

Are Pretrained Convolutions Better than Pretrained Transformers?

MLP-Mixer: An all-MLP Architecture for Vision

Random Feature Attention

Do Transformer Modifications Transfer Across Implementations and Applications?

Rethinking FUN: Frequency-Domain Utilization Networks

Language Through a Prism: A Spectral Approach for Multiscale Language Representations

Long Range Arena: A Benchmark for Efficient Transformers

FastFormers: Highly Efficient Transformer Models for Natural Language Understanding

Pre-trained Summarization Distillation

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Fourier Neural Operator for Parametric Partial Differential Equations

Rethinking Attention with Performers

Efficient Transformers: A Survey

Big Bird: Transformers for Longer Sequences

Fast Transformers with Clustered Attention

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Linformer: Self-Attention with Linear Complexity

Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers

Language Models are Few-Shot Learners

Synthesizer: Rethinking Self-Attention for Transformer Models

Hard-Coded Gaussian Attention for Neural Machine Translation

Synthesizer: Rethinking Self-Attention in Transformer Models

ETC: Encoding Long and Structured Inputs in Transformers

Longformer: The Long-Document Transformer

Acceleration of Convolutional Neural Network Using FFT-Based Split Convolutions

Efficient Content-Based Sparse Attention with Routing Transformers

Sparse Sinkhorn Attention

Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

Reformer: The Efficient Transformer

Blockwise Self-Attention for Long Document Understanding

Fast Transformer Decoding: One Write-Head is All You Need

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

TinyBERT: Distilling BERT for Natural Language Understanding

Designing and Interpreting Probes with Control Tasks

What Does BERT Look at? An Analysis of BERT’s Attention

Analyzing the Structure of Attention in a Transformer Language Model

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

BERT Rediscovers the Classical NLP Pipeline

Generating Long Sequences with Sparse Transformers

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Learning Long Term Dependencies via Fourier Recurrent Units

Image Transformer

Generating Wikipedia by Summarizing Long Sequences

FFT-based deep learning deployment in embedded systems

FCNN: Fourier Convolutional Neural Networks

Attention is All you Need

Very Efficient Training of Convolutional Neural Networks using Fast Fourier Transform and Overlap-and-Add

ACDC: A Structured Efficient Linear Layer

Structured Transforms for Small-Footprint Deep Learning

Fast fourier transform for feature extraction and neural network for classification of electrocardiogram signals

An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections

Neural Machine Translation by Jointly Learning to Align and Translate

Fast Training of Convolutional Networks through FFTs

Fault diagnosis and prognosis using wavelet packet decomposition, Fourier transform and artificial neural network

Cardiac arrhythmias detection in an ECG beat signal using fast fourier transform and artificial neural network

Random Features for Large-Scale Kernel Machines

Fast Object/Face Detection Using Neural Networks and Fast Fourier Transform

The Design and Implementation of FFTW3

Real-time discrimination of ventricular tachyarrhythmia with Fourier-transform neural network

Using Fourier-neural recurrent networks to fit sequential input/output data

Universal approximation bounds for superpositions of a sigmoidal function

Approximation by superpositions of a sigmoidal function

On the Equivalence Between One-Dimensional Discrete Walsh-Hadamard and Multidimensional Discrete Fourier Transforms

An algorithm for the machine calculation of complex Fourier series

A note on more eﬃcient architectures for NLP

2021a). We also perform a very small sweep over the embedding dimension and batch

Data Movement Is All You Need: A Case Study of Transformer Networks

FNet’s mixing weights, on

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Fast Fourier Transformed Transformers: Circulant Weight Matrices for NMT Compression

Linformer (Wang et al., 2020) and Linear Transformer (Katharopoulos et al., 2020)