Online and Linear-Time Attention by Enforcing Monotonic Alignments (2017-04-03T00:00:00.000000Z)

TL;DR

This work proposes an end-to-end differentiable method for learning monotonic alignments which, at test time, enables computing attention online and in linear time and validates the approach on sentence summarization, machine translation, and online speech recognition problems.

Abstract

Recurrent neural network models with an attention mechanism have proven to be extremely effective on a wide variety of sequence-to-sequence problems. However, the fact that soft attention mechanisms perform a pass over the entire input sequence when producing each element in the output sequence precludes their use in online settings and results in a quadratic time complexity. Based on the insight that the alignment between input and output sequence elements is monotonic in many problems of interest, we propose an end-to-end differentiable method for learning monotonic alignments which, at test time, enables computing attention online and in linear time. We validate our approach on sentence summarization, machine translation, and online speech recognition problems and achieve results competitive with existing sequence-to-sequence models.

Authors

Peter J. Liu

7 papers

D. Eck

8 papers

Ron J. Weiss

14 papers

TL;DR

Abstract

Authors

References61 items

Training a Subsampling Mechanism in Expectation

Structured Attention Networks

Cutting-off Redundant Repeating Generations for Neural Abstractive Summarization

Towards Better Decoding and Language Model Integration in Sequence to Sequence Models

Efficient Summarization with Read-Again and Copy Mechanism

The Neural Noisy Channel

Sequence to Sequence Transduction with Hard Monotonic Attention

Categorical Reparameterization with Gumbel-Softmax

The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables

Hybrid computing using a neural network with dynamic external memory

Very deep convolutional networks for end-to-end speech recognition

Online Segment to Segment Neural Transduction

Language as a Latent Variable: Discrete Generative Models for Sentence Compression

Learning online alignments with continuous rewards policy gradient

Abstractive Sentence Summarization with Attentive Recurrent Neural Networks

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

Adaptive Computation Time for Recurrent Neural Networks

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

Lookahead Convolution Layer for Unidirectional Recurrent Neural Networks

Learning Efficient Algorithms with Hierarchical Attentive Memory

Neural GPUs Learn Algorithms

Neural Random Access Machines

Segmental Recurrent Neural Networks

A Neural Transducer

A Neural Attention Model for Abstractive Sentence Summarization

Effective Approaches to Attention-based Neural Machine Translation

Listen, attend and spell: A neural network for large vocabulary conversational speech recognition

Grid Long Short-Term Memory

Attention-Based Models for Speech Recognition

Learning to Transduce with Unbounded Memory

Reinforcement Learning Neural Turing Machines

End-To-End Memory Networks

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Adam: A Method for Stochastic Optimization

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Neural Turing Machines

Sequence to Sequence Learning with Neural Networks

Neural Machine Translation by Jointly Learning to Align and Translate

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Dropout Improves Recurrent Neural Networks for Handwriting Recognition

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

Generating Sequences With Recurrent Neural Networks

Speech recognition with deep recurrent neural networks

Sequence Transduction with Recurrent Neural Networks

Semantic hashing

Long Short-Term Memory

Continuous Sigmoidal Belief Networks Trained using Slice Sampling

On Parallel Prefix Computation

The Design for the Wall Street Journal-based CSR Corpus

Online and Linear-Time Attention by Enforcing Monotonic Alignments

Text summarization with TensorFlow

Stanford Neural Machine Translation Systems for Spoken Language Domains

The IWSLT 2015 Evaluation Campaign

Eigen v3

DARPA TIMIT acoustic-phonetic continous speech

Connectionist Temporal Classiﬁcation: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

This Paper Is Included in the Proceedings of the 12th Usenix Symposium on Operating Systems Design and Implementation (osdi '16). Tensorflow: a System for Large-scale Machine Learning Tensorflow: a System for Large-scale Machine Learning

Connectionist Temporal Classiﬁcation: Labelling Unsegmented Sequences with Recurrent Neural Networks

Field of Study

Venue Information

Name

Type

URL

Alternate Names