Single Headed Attention RNN: Stop Thinking With Your Head

Published in

arXiv.org(2019)

External Links:

Generate Graph

TL;DR

This work has undergone no intensive hyperparameter optimization and lived entirely on a commodity desktop machine that made the author's small studio apartment far too warm in the midst of a San Franciscan summer.

Abstract

The leading approaches in language modeling are all obsessed with TV shows of my youth - namely Transformers and Sesame Street. Transformers this, Transformers that, and over here a bonfire worth of GPU-TPU-neuromorphic wafer scale silicon. We opt for the lazy path of old and proven techniques with a fancy crypto inspired acronym: the Single Headed Attention RNN (SHA-RNN). The author's lone goal is to show that the entire field might have evolved a different direction if we had instead been obsessed with a slightly different acronym and slightly different result. We take a previously strong language model based only on boring LSTMs and get it to within a stone's throw of a stone's throw of state-of-the-art byte level language model results on enwik8. This work has undergone no intensive hyperparameter optimization and lived entirely on a commodity desktop machine that made the author's small studio apartment far too warm in the midst of a San Franciscan summer. The final results are achievable in plus or minus 24 hours on a single GPU as the author is impatient. The attention mechanism is also readily extended to large contexts with minimal computation. Take that Sesame Street.

Authors

Stephen Merity

4 papers

References22 items

Fast Transformer Decoding: One Write-Head is All You Need

Adaptive Attention Span in Transformers

Generating Long Sequences with Sparse Transformers

Reducing BERT Pre-Training Time from 3 Days to 76 Minutes

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Single Headed Attention RNN: Stop Thinking With Your Head

Published in

arXiv.org(2019)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Stephen Merity

4 papers

TL;DR

Abstract

Authors

References22 items

Fast Transformer Decoding: One Write-Head is All You Need

Adaptive Attention Span in Transformers

Generating Long Sequences with Sparse Transformers

Reducing BERT Pre-Training Time from 3 Days to 76 Minutes

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

TL;DR

Abstract

Authors

References22 items

Fast Transformer Decoding: One Write-Head is All You Need

Adaptive Attention Span in Transformers

Generating Long Sequences with Sparse Transformers

Reducing BERT Pre-Training Time from 3 Days to 76 Minutes

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Character-Level Language Modeling with Deeper Self-Attention

An Analysis of Neural Language Modeling at Multiple Scales

Regularizing and Optimizing LSTM Language Models

Attention is All you Need

Improving Neural Language Models with a Continuous Cache

Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

Pointer Sentinel Mixture Models

Character-level language modeling with hierarchical recurrent neural networks

Using the Output Embedding to Improve Language Models

Recurrent Highway Networks

Long Short-Term Memory

Finding Structure in Time

Lossless Data Compression with Neural Networks

Mogriﬁer LSTM

The Human Knowledge Compression Contest

SUBWORD LANGUAGE MODELING WITH NEURAL NETWORKS

, Tomáš , and Blunsom , Phil . Mogri - fier LSTM . arXiv preprint arXiv : 1909

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names