Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (2018-09-27T00:00:00.000000Z)

TL;DR

The novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

Abstract

Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such an inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.

Authors

Aaron C. Courville

35 papers

Alessandro Sordoni

8 papers

Shawn Tan

2 papers

TL;DR

Abstract

Authors

References69 items

On Aspects of the Theory of Syntax

The emergence of number and syntax units in LSTM language models

Grammar Induction with Neural Language Models: An Unusual Replication

On Tree-Based Neural Sentence Modeling

Targeted Syntactic Evaluation of Language Models

Learning Hierarchical Structures On-The-Fly with a Recurrent-Recursive Model for Sequences

LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better

Colorless Green Recurrent Networks Dream Hierarchically

Memory Architectures in Recurrent Neural Network Language Models

The Neural Network Pushdown Automaton: Model, Stack and Learning Simulations

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

Neural Language Modeling by Jointly Learning Syntax and Lexicon

Do latent tree learning models identify meaningful structure in sentences?

Regularizing and Optimizing LSTM Language Models

On the State of the Art of Evaluation in Neural Language Models

Learning to Compose Task-Specific Tree Structures

Sequence-to-Dependency Neural Machine Translation

Generative Neural Machine for Tree Structures

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Learning to Compose Words into Sentences with Reinforcement Learning

Neural Architecture Search with Reinforcement Learning

Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

Improving Neural Language Models with a Continuous Cache

Tree-structured decoding with doubly-recurrent neural networks

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

Pointer Sentinel Mixture Models

Hierarchical Multiscale Recurrent Neural Networks

Using the Output Embedding to Improve Language Models

Recurrent Highway Networks

A Fast Unified Model for Parsing and Sentence Understanding

Recurrent Neural Network Grammars

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Top-down Tree Long Short-Term Memory Networks

The Neural Representation of Sequences: From Transition Probabilities to Algebraic Patterns and Linguistic Trees

Character-Aware Neural Language Models

Gradient Estimation Using Stochastic Computation Graphs

Tree-Structured Composition in Neural Networks without Tree-Structured Architectures

Learning to Transduce with Unbounded Memory

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

Recurrent Neural Network Regularization

Recursive Neural Networks Can Learn Logical Semantics

Deep learning in neural networks: An overview

A Clockwork RNN

Learning Ordered Representations with Nested Dropout

An Introduction to Syntactic Analysis and Theory

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Unsupervised Structure Prediction with Non-Parallel Multilingual Guidance

An All-Subtrees Approach to Unsupervised Parsing

Natural language grammar induction with a generative constituent-context model

Accurate Unlexicalized Parsing

A Generative Constituent-Context Model for Improved Grammar Induction

LSTM recurrent networks learn simple context-free and context-sensitive languages

Immediate-Head Parsing for Language Models

Probabilistic Top-Down Parsing and Language Modeling

Structured language modeling

Morphological structure, lexical representation and lexical access

Hierarchical Recurrent Neural Networks for Long-Term Dependencies

Learning long-term dependencies is not as difficult with NARX recurrent neural networks

Bayesian Grammar Induction for Language Modeling

Productivity in language production

Building a Large Annotated Corpus of English: The Penn Treebank

Three models for the description of language

Deep Learning

Statistical Language Models Based on Neural Networks

Learning Continuous Phrase Representations and Syntactic Parsing with Recursive Neural Networks

Learning Deep Architectures for AI

Neural sequence chunkers

a decision is n’t expected until some time next year A decision is n’t expected until some time next year Figure A.1:

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type