Compound Probabilistic Context-Free Grammars for Grammar Induction (2019-06-24T00:00:00.000000Z)

TL;DR

A formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context free grammar, which is modulated by a per-sentence continuous latent variable, which induces marginal dependencies beyond the traditional context-free assumptions.

Abstract

We study a formalization of the grammar induction problem that models sentences as being generated by a compound probabilistic context free grammar. In contrast to traditional formulations which learn a single stochastic grammar, our context-free rule probabilities are modulated by a per-sentence continuous latent variable, which induces marginal dependencies beyond the traditional context-free assumptions. Inference in this context-dependent grammar is performed by collapsed variational inference, in which an amortized variational posterior is placed on the continuous variable, and the latent trees are marginalized with dynamic programming. Experiments on English and Chinese show the effectiveness of our approach compared to recent state-of-the-art methods for grammar induction from words with neural language models.

Authors

Chris Dyer

16 papers

Alexander M. Rush

18 papers

Yoon Kim

5 papers

TL;DR

Abstract

Authors

References92 items

Visually Grounded Neural Syntax Acquisition

An Imitation Learning Approach to Unsupervised Parsing

Controllable Paraphrase Generation with a Syntactic Exemplar

Scalable Syntax-Aware Language Models Using Knowledge Distillation

Text Generation with Exemplar-based Adaptive Decoding

Unsupervised Recurrent Neural Network Grammars

Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders

A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations

Structural Supervision Improves Learning of Non-Local Grammatical Dependencies

Structured Alignment Networks for Matching Sentences

Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Benefits of over-parameterization with EM

Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

Grammar Induction with Neural Language Models: An Unusual Replication

Targeted Syntactic Evaluation of Language Models

Learning Neural Templates for Text Generation

Unsupervised Learning of Syntactic Structure with Invertible Neural Projections

LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better

Constituency Parsing with a Self-Attentive Encoder

Gaussian Mixture Latent Vector Grammars

Mutual Information Maximization for Simple and Accurate Part-Of-Speech Induction

Unsupervised Grammar Induction with Depth-bounded PCFG

On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization

Neural Language Modeling by Jointly Learning Syntax and Lexicon

A Minimal Span-Based Neural Constituency Parser

Inside-Outside and Forward-Backward Algorithms Are Just Backprop (tutorial paper)

Unsupervised Neural Hidden Markov Models

Using the Output Embedding to Improve Language Models

Graph-based Dependency Parsing with Bidirectional LSTM

Layer Normalization

Improved Variational Inference with Inverse Autoregressive Flow

Bayesian Analysis in Natural Language Processing

Exploring the Limits of Language Modeling

Recurrent Neural Network Grammars

Character-Aware Neural Language Models

Neural CRF Parsing

Variational Inference with Normalizing Flows

Long Short-Term Memory Over Tree Structures

Unsupervised POS Induction with Word Embeddings

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

Adam: A Method for Stochastic Optimization

Learning Character-level Representations for Part-of-Speech Tagging

Spectral Unsupervised Parsing with Additive Tree Metrics

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

Breaking Out of Local Optima with Count Transforms and Model Recombination: A Study in Grammar Induction

Collapsed Variational Bayesian Inference for PCFGs

Parsing with Compositional Vector Grammars

Improved Constituent Context Model with Features

Three Dependency-and-Boundary Models for Grammar Induction

A Feature-Rich Constituent Context Model for Grammar Induction

Spectral Learning of Latent-Variable PCFGs

Improved Fully Unsupervised Parsing with Zoomed Learning

Painless Unsupervised Learning with Features

Unsupervised Multilingual Grammar Induction

Curriculum learning

Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction

Online EM for Unsupervised Models

Logistic Normal Priors for Unsupervised Probabilistic Grammar Induction

Fast Unsupervised Incremental Parsing

The Infinite PCFG Using Hierarchical Dirichlet Processes

Bayesian Inference for PCFGs via Markov Chain Monte Carlo

Variational Bayesian Grammar Induction for Natural Language

An All-Subtrees Approach to Unsupervised Parsing

Learning Accurate, Compact, and Interpretable Tree Annotation

Probabilistic CFG with Latent Annotations

The Penn Chinese TreeBank: Phrase structure annotation of a large corpus

Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency

Annealing Techniques For Unsupervised Statistical Language Learning

Optimization with EM and Expectation-Conjugate-Gradient

Accurate Unlexicalized Parsing

Compound decision theory and empirical bayes methods

A Generative Constituent-Context Model for Improved Grammar Induction

Unsupervised induction of stochastic context-free grammars using distributional clustering

PCFG Models of Linguistic Tree Representations

Three Generative, Lexicalised Models for Statistical Parsing