Learning to Transduce with Unbounded Memory
Published in
Neural Information Processing Systems(2015)External Links:
TL
TL;DR
This paper proposes new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues and shows that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in the transduction experiments.
Abstract
Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.
References22 items
1
The Neural Network Pushdown Automaton: Model, Stack and Learning Simulations
2
Transition-Based Dependency Parsing with Stack Long Short-Term Memory
3
Reinforcement Learning Neural Turing Machines
4
Reinforcement Learning Neural Turing Machines - Revised
5