A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models (2016-03-01T00:00:00.000000Z)

TL;DR

A novel latent variable recurrent neural network architecture for jointly modeling sequences of words and (possibly latent) discourse relations between adjacent sentences that outperforms state-of-the-art alternatives for two tasks: implicit discourse relation classification in the Penn Discourse Treebank, and dialog act Classification in the Switchboard corpus.

Abstract

This paper presents a novel latent variable recurrent neural network architecture for jointly modeling sequences of words and (possibly latent) discourse relations between adjacent sentences. A recurrent neural network generates individual words, thus reaping the benefits of discriminatively-trained vector representations. The discourse relations are represented with a latent variable, which can be predicted or marginalized, depending on the task. The resulting model can therefore employ a training objective that includes not only discourse relation classification, but also word prediction. As a result, it outperforms state-of-the-art alternatives for two tasks: implicit discourse relation classification in the Penn Discourse Treebank, and dialog act classification in the Switchboard corpus. Furthermore, by marginalizing over latent discourse relations at test time, we obtain a discourse informed language model, which improves over a strong LSTM baseline.

Authors

Jacob Eisenstein

7 papers

Gholamreza Haffari

8 papers

Yangfeng Ji

6 papers

TL;DR

Abstract

Authors

References56 items

Document Context Language Models

Larger-Context Language Modelling with Recurrent Neural Network

Comparing Word Representations for Implicit Discourse Relation Classification

Importance Weighted Autoencoders

Hierarchical Recurrent Neural Network for Document Modeling

Shallow Convolutional Neural Network for Implicit Discourse Relation Recognition

The CoNLL-2015 Shared Task on Shallow Discourse Parsing

A Neural Network Approach to Context-Sensitive Generation of Conversational Responses

Neural Adaptive Sequential Monte Carlo

One Vector is Not Enough: Entity-Augmented Distributed Semantics for Discourse Relations

A Recurrent Latent Variable Model for Sequential Data

Transition-Based Dependency Parsing with Stack Long Short-Term Memory

Recursive Deep Models for Discourse Parsing

Recurrent Neural Network Regularization

Representation Learning for Text-level Discourse Parsing

Proceedings of the Association for Computational Linguistics 2014

Distributed Representations of Sentences and Documents

Discovering Implicit Discourse Relations Through Brown Cluster Pair Representation and Coreference Patterns

Investigating the Contribution of Distributional Semantic Information for Dialogue Act Classification

Auto-Encoding Variational Bayes

Dropout Improves Recurrent Neural Networks for Handwriting Recognition

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Aggregated Word Pair Features for Implicit Discourse Relation Disambiguation

Recurrent Convolutional Neural Networks for Discourse Compositionality

Context dependent recurrent neural network language model

On the difficulty of training recurrent neural networks

A Novel Discriminative Framework for Sentence-Level Discourse Analysis

Discourse Structure and Computation: Past, Present and Future

Improving Implicit Discourse Relation Recognition Through Feature Set Optimization

Practical Recommendations for Gradient-Based Training of Deep Architectures

Discourse structure and language technology

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

Natural Language Processing (Almost) from Scratch

Word Representations: A Simple and General Method for Semi-Supervised Learning

Analysis of Discourse Structure with Syntactic Dependencies and Data-Driven Shift-Reduce Parsing

Recognizing Implicit Discourse Relations in the Penn Discourse Treebank

Automatic sense prediction for implicit discourse relations in text

Easily Identifiable Discourse Relations

The Penn Discourse TreeBank 2.0.

Partially observable Markov decision processes for spoken dialog systems

Solving the Problem of Cascading Errors: Approximate Bayesian Inference for Linguistic Annotation Pipelines

Dialogue act modeling for automatic tagging and recognition of conversational speech

Sequential Monte Carlo Methods to Train Neural Network Models

Long Short-Term Memory

Multitask Learning

SWITCHBOARD: telephone speech corpus for research and development

Computational Linguistics and Deep Learning

Improving the Inference of Implicit Discourse Relations via Classifying Explicit Discourse Connectives

employ a recursive neural network architecture

We use the “context-to-output” variant, which is identical to the current modeling approach, except that it is not parametrized by discourse relations

Dropout: a simple way to prevent neural networks from overfitting

Recurrent neural network based language model

Dialogue Act Classification Based on Intra-Utterance Features∗

employ a hidden Markov model, with each HMM state corresponding to a dialogue act

Rich Caruana. 1997. Multitask learning. Machine learning

Proceedings of Empirical Methods for Natural Language Processing (EMNLP)

Field of Study

Venue Information

Name

Type

URL

Alternate Names