Neural Language Modeling by Jointly Learning Syntax and Lexicon (2017-11-01T00:00:00.000000Z)

TL;DR

A novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model is proposed.

Abstract

We propose a neural language model capable of unsupervised syntactic structure induction. The model leverages the structure information to form better semantic representations and better language modeling. Standard recurrent neural networks are limited by their structure and fail to efficiently use syntactic information. On the other hand, tree-structured recursive networks usually require additional structural supervision at the cost of human expert annotation. In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model. In our model, the gradient can be directly back-propagated from the language model loss into the neural parsing network. Experiments show that the proposed model can discover the underlying syntactic structure and achieve state-of-the-art performance on word/character-level language model tasks.

Authors

Aaron C. Courville

35 papers

Zhouhan Lin

9 papers

Yikang Shen

3 papers

TL;DR

Abstract

Authors

References66 items

Do latent tree learning models identify meaningful structure in sentences?

Learning to parse from a semantic objective: It works. Is it syntax?

On the State of the Art of Evaluation in Neural Language Models

Sequence-to-Dependency Neural Machine Translation

Generative Neural Machine for Tree Structures

What Do Recurrent Neural Network Grammars Learn About Syntax?

Improving Neural Language Models with a Continuous Cache

Neural Architecture Search with Reinforcement Learning

Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

Tree-structured decoding with doubly-recurrent neural networks

An Evaluation of Parser Robustness for Ungrammatical Sentences

HyperNetworks

Pointer Sentinel Mixture Models

Hierarchical Multiscale Recurrent Neural Networks

Using the Output Embedding to Improve Language Models

Layer Normalization

Recurrent Highway Networks

On Multiplicative Integration with Recurrent Neural Networks

Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations

Recurrent Batch Normalization

A Fast Unified Model for Parsing and Sentence Understanding

Recurrent Neural Network Grammars

Long Short-Term Memory-Networks for Machine Reading

Regularizing RNNs by Stabilizing Activations

Top-down Tree Long Short-Term Memory Networks

Character-Aware Neural Language Models

Generative Incremental Dependency Parsing with Neural Networks

End-To-End Memory Networks

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

Learning Longer Memory in Recurrent Neural Networks

Recurrent Neural Network Regularization

Deep learning in neural networks: An overview

A Clockwork RNN

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Efficient Estimation of Word Representations in Vector Space

The Expressive Power of Word Embeddings

Context dependent recurrent neural network language model

Classifying Chart Cells for Quadratic Complexity Context-Free Inference

A Latent Variable Model for Generative Dependency Parsing

An All-Subtrees Approach to Unsupervised Parsing

Natural language grammar induction with a generative constituent-context model

A Neural Syntactic Language Model

Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency

Accurate Unlexicalized Parsing

A Neural Probabilistic Language Model

A Generative Constituent-Context Model for Improved Grammar Induction

Unsupervised induction of stochastic context-free grammars using distributional clustering

Immediate-Head Parsing for Language Models

Probabilistic Top-Down Parsing and Language Modeling

Morphological structure, lexical representation and lexical access

Long Short-Term Memory

A Structured Language Model

Hierarchical Recurrent Neural Networks for Long-Term Dependencies

Learning long-term dependencies is not as difficult with NARX recurrent neural networks

Building a Large Annotated Corpus of English: The Penn Treebank

Two Experiments on Learning Probabilistic Dependency Grammars from Corpora

Twelve Years of Unsupervised Dependency Parsing

Deep Learning

Aspects of the Theory of Syntax , volume 11

Large text compression benchmark

SUBWORD LANGUAGE MODELING WITH NEURAL NETWORKS

Learning Continuous Phrase Representations and Syntactic Parsing with Recursive Neural Networks

Recurrent neural network based language model

Learning Deep Architectures for AI

Automatic Acquisition and Efficient Representation of Syntactic Structures

Neural sequence chunkers

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names