Mega: Moving Average Equipped Gated Attention (2022-09-21T00:00:00.000000Z)

TL;DR

This paper introduces Mega, a simple, theoretically grounded, single-head gated attention mechanism equipped with (exponential) moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism.

Abstract

The design choices in the Transformer attention mechanism, including weak inductive bias and quadratic computational complexity, have limited its application for modeling long sequences. In this paper, we introduce Mega, a simple, theoretically grounded, single-head gated attention mechanism equipped with (exponential) moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism. We further propose a variant of Mega that offers linear time and space complexity yet yields only minimal quality loss, by efficiently splitting the whole sequence into multiple chunks with fixed length. Extensive experiments on a wide range of sequence modeling benchmarks, including the Long Range Arena, neural machine translation, auto-regressive language modeling, and image and speech classification, show that Mega achieves significant improvements over other sequence models, including variants of Transformers and recent state space models.

Authors

Graham Neubig

38 papers

Chunting Zhou

5 papers

Junxian He

5 papers

TL;DR

Abstract

Authors

References81 items

Long Range Language Modeling via Gated State Spaces

On the Parameterization and Initialization of Diagonal State Space Models

Towards a General Purpose CNN for Long Range Dependencies in ND

Transformer Quality in Linear Time

Efficiently Modeling Long Sequences with Structured State Spaces

FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes

Primer: Searching for Efficient Transformers for Language Modeling

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Highly accurate protein structure prediction with AlphaFold

CoAtNet: Marrying Convolution and Attention for All Data Sizes

Luna: Linear Unified Nested Attention

Pay Attention to MLPs

RoFormer: Enhanced Transformer with Rotary Position Embedding

Random Feature Attention

When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute

CKConv: Continuous Kernel Convolution For Sequential Data

Training data-efficient image transformers & distillation through attention

Long Range Arena: A Benchmark for Efficient Transformers

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Rethinking Attention with Performers

Improve Transformer Models with Better Relative Position Embeddings

Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization

Efficient Transformers: A Survey

HiPPO: Recurrent Memory with Optimal Polynomial Projections

Rewiring the Transformer with Depth-Wise LSTMs

Rethinking Positional Encoding in Language Pre-training

Minimum Width for Universal Approximation

Linformer: Self-Attention with Linear Complexity

Augment Your Batch: Improving Generalization Through Instance Repetition

Longformer: The Long-Document Transformer

ProGen: Language Modeling for Protein Generation

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Reformer: The Efficient Transformer

Compressive Transformers for Long-Range Sequence Modelling

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Stabilizing Transformers for Reinforcement Learning

Randaugment: Practical automated data augmentation with a reduced search space

RoBERTa: A Robustly Optimized BERT Pretraining Approach

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Adaptive Input Representations for Neural Language Modeling

Character-Level Language Modeling with Deeper Self-Attention

Scaling Neural Machine Translation

Learning long-range spatial dependencies with horizontal gated-recurrent units

A Call for Clarity in Reporting BLEU Scores

Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

ListOps: A Diagnostic Dataset for Latent Tree Learning

Image Transformer

mixup: Beyond Empirical Risk Minimization

Swish: a Self-Gated Activation Function

Random Erasing Data Augmentation

Attention is All you Need

Convolutional Sequence to Sequence Learning

Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning

Fast and Accurate Entity Recognition with Iterated Dilated Convolutions

Error bounds for approximations with deep ReLU networks

Pointer Sentinel Mixture Models

Deep Networks with Stochastic Depth

Complex exponential smoothing

Neural Machine Translation of Rare Words with Subword Units

Effective Approaches to Attention-based Neural Machine Translation

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Neural Machine Translation by Jointly Learning to Align and Translate

Convolutional Neural Networks for Sentence Classification

Findings of the 2014 Workshop on Statistical Machine Translation

Learning Word Vectors for Sentiment Analysis

Damped trend exponential smoothing: A modelling viewpoint

The Exponentially Weighted Moving Average

The ACL anthology network corpus

ImageNet: A large-scale hierarchical image database

Long Short-Term Memory

Multilayer feedforward networks are universal approximators

Forecasting Sales by Exponentially Weighted Moving Averages

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

2022a), we perform speech classification on the SC10 subset of the Speech Commands dataset (Warden, 2018), which is a 10-class classification task. The chunk size of Mega-chunk is 1000