Compressive Transformers for Long-Range Sequence Modelling (2019-11-13T00:00:00.000000Z)

TL;DR

The Compressive Transformer is presented, an attentive sequence model which compresses past memories for long-range sequence learning and can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task.

Abstract

We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.

Authors

T. Lillicrap

23 papers

Jack W. Rae

3 papers

Anna Potapenko

2 papers

TL;DR

Abstract

Authors

References49 items

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Large Memory Layers with Product Keys

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Adaptive Attention Span in Transformers

Generating Long Sequences with Sparse Transformers

The Curious Case of Neural Text Degeneration

Dynamic Evaluation of Transformer Language Models

Pay Less Attention with Lightweight and Dynamic Convolutions

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Adaptive Input Representations for Neural Language Modeling

Trellis Networks for Sequence Modeling

Character-Level Language Modeling with Deeper Self-Attention

Relational recurrent neural networks

End-to-End Dense Video Captioning with Masked Transformer

Fast Parametric Learning with Activation Memorization

Convolutional Sequence Modeling Revisited

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

The NarrativeQA Reading Comprehension Challenge

Parallel WaveNet: Fast High-Fidelity Speech Synthesis

Don't Decay the Learning Rate, Increase the Batch Size

The Persistence and Transience of Memory

Attention is All you Need

Language Modeling with Gated Convolutional Networks

Quasi-Recurrent Neural Networks

Improving Neural Language Models with a Continuous Cache

Neural Machine Translation in Linear Time

Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes

Hybrid computing using a neural network with dynamic external memory

Pointer Sentinel Mixture Models

Multiplicative LSTM for sequence modelling

WaveNet: A Generative Model for Raw Audio

Hierarchical Multiscale Recurrent Neural Networks

Recurrent Highway Networks

The LAMBADA dataset: Word prediction requiring a broad discourse context

The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Adam: A Method for Stochastic Optimization

Neural Turing Machines

Neural Machine Translation by Jointly Learning to Align and Translate

One billion word benchmark for measuring progress in statistical language modeling

Generating Sequences With Recurrent Neural Networks

Long Short-Term Memory

Learning representations by back-propagating errors

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

The human knowledge compression contest

Recurrent neural network based language model

Latent Dirichlet Allocation

OMPETING INTERESTS The authors declare no competing ﬁnancial interests

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names