Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Auto-Encoders (2019-04-03T00:00:00.000000Z)

TL;DR

DIORA is introduced, a fully-unsupervised method for discovering syntax that simultaneously learns representations for constituents within the induced tree that outperforms previously reported results for unsupervised binary constituency parsing on the benchmark WSJ dataset.

Abstract

We introduce the deep inside-outside recursive autoencoder (DIORA), a fully-unsupervised method for discovering syntax that simultaneously learns representations for constituents within the induced tree. Our approach predicts each word in an input sentence conditioned on the rest of the sentence. During training we use dynamic programming to consider all possible binary trees over the sentence, and for inference we use the CKY algorithm to extract the highest scoring parse. DIORA outperforms previously reported results for unsupervised binary constituency parsing on the benchmark WSJ dataset.

Authors

A. McCallum

17 papers

Mohit Iyyer

16 papers

Pat Verga

4 papers

TL;DR

Abstract

Authors

References60 items

VOCABULARY

Structured Alignment Networks for Matching Sentences

Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

Grammar Induction with Neural Language Models: An Unusual Replication

Dissecting Contextual Word Embeddings: Architecture and Representation

Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling

Deep Contextualized Word Representations

STRUCTURED ALIGNMENT NETWORKS

Incorporating Syntactic Uncertainty in Neural Machine Translation with a Forest-to-Sequence Model

Neural Language Modeling by Jointly Learning Syntax and Lexicon

Do latent tree learning models identify meaningful structure in sentences?

Learning to Compose Task-Specific Tree Structures

Jointly learning sentence embeddings and syntax with unsupervised Tree-LSTMs

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Towards String-To-Tree Neural Machine Translation

Learning to Parse and Translate Improves Neural Machine Translation

Learning to Compose Words into Sentences with Reinforcement Learning

The Forest Convolutional Network: Compositional Distributional Semantics with a Neural Chart and without Binarization

A large annotated corpus for learning natural language inference

Neural CRF Parsing

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

Adam: A Method for Stochastic Optimization

The Inside-Outside Recursive Neural Network model for Dependency Parsing

The Stanford CoreNLP Natural Language Processing Toolkit

Breaking Out of Local Optima with Count Transforms and Model Recombination: A Study in Grammar Induction

CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes

Dependency-Based Open Information Extraction

Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Unsupervised Structure Prediction with Non-Parallel Multilingual Guidance

Simple Unsupervised Grammar Induction from Raw Text with Cascaded Finite State Models

Unsupervised Part-of-Speech Tagging with Bilingual Graph-Based Projections

A Bayesian framework for word segmentation: Exploring the effects of context

An All-Subtrees Approach to Unsupervised Parsing

Sense and Reference

Learning question classifiers: the role of semantic information

Joint Parsing and Semantic Role Labeling

Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency

A Generative Constituent-Context Model for Improved Grammar Induction

Introduction to the CoNLL-2000 Shared Task Chunking

Mostly-Unsupervised Statistical Segmentation of Japanese: Applications to Kanji

The Penn Treebank: Annotating Predicate Argument Structure

Building a Large Annotated Corpus of English: The Penn Treebank

Deducing Linguistic Structure from the Statistics of Large Corpora

Trainable grammars for speech recognition

Recognition and Parsing of Context-Free Languages in Time n^3

An Efficient Recognition and Syntax-Analysis Algorithm for Context-Free Languages

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

2018a. Do latent tree learning models identify meaningful structure in sentences? Transactions of the Association of Computational Linguistics (TACL), 6:253–267

Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration

2018b. Ordered neurons: Integrat

Composition and Input Transform TreeLSTM. The TreeLSTM (Tai et al., 2015) function produces a hidden state vector h and cell state vector c given two input vectors hi and hj

The estimation of stochastic context-free grammars using the Inside-Outside algorithm

Applications of stochastic context-free grammars using the Inside-Outside algorithm

the outside. U and b are learned

A.2 Composition and Input Transform TreeLSTM . The TreeLSTM

is taken from ELMo (Peters et al.,

Sampling . N negatives are sampled for each batch. All experiments use N = 100 . Training Steps . 1M parameter updates, taking 3 days using 4x

workshop on unsupervised and semi-supervised learning

o u     σ tanh   (cid:20) j (cid:21)  0 0   c = c i ⊙ f i + c j ⊙ f j + x ⊙ u h = o + tanh( c ) The constant ω is set to 1 for the inside,

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names