Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer (2019-05-01T00:00:00.000000Z)

TL;DR

A novel regularizer is developed to improve the learning of long-range dependency of sequence data and shows how the `next sentence prediction (classification)' heuristic can be derived in a principled way from the authors' mutual information estimation framework, and be further extended to maximize the mutual information of sequence variables.

Abstract

In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex long-range dependency. We show how the `next sentence prediction (classification)' heuristic can be derived in a principled way from our mutual information estimation framework, and be further extended to maximize the mutual information of sequence variables. The proposed approach not only is effective at increasing the mutual information of segments under the learned model but more importantly, leads to a higher likelihood on holdout data, and improved generation quality.

Authors

Peng Xu

3 papers

Yanshuai Cao

1 papers

TL;DR

Abstract

Authors

References42 items

ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

RoBERTa: A Robustly Optimized BERT Pretraining Approach

SpanBERT: Improving Pre-training by Representing and Predicting Spans

XLNet: Generalized Autoregressive Pretraining for Language Understanding

A Cross-Domain Transferable Neural Coherence Model

ERNIE: Enhanced Language Representation with Informative Entities

What comes next? Extractive summarization by next-sentence prediction

Learning deep representations by mutual information estimation and maximization

Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN

MINE: Mutual Information Neural Estimation

Learning Longer-term Dependencies in RNNs with Auxiliary Losses

Topic Compositional Neural Language Model

Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

Regularizing and Optimizing LSTM Language Models

Adversarially Regularized Autoencoders

Attention is All you Need

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Self-Critical Sequence Training for Image Captioning

TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency

Pointer Sentinel Mixture Models

Reward Augmented Maximum Likelihood for Neural Structured Prediction

On Multiplicative Integration with Recurrent Neural Networks

Larger-Context Language Modelling with Recurrent Neural Network

Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks

A Simple Way to Initialize Recurrent Networks of Rectified Linear Units

End-To-End Memory Networks

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

Learning Longer Memory in Recurrent Neural Networks

Adam: A Method for Stochastic Optimization

Neural Machine Translation by Jointly Learning to Align and Translate

A Clockwork RNN

Context dependent recurrent neural network language model

Learning Recurrent Neural Networks with Hessian-Free Optimization

Introduction to Nonparametric Estimation

Elements of Information Theory

A Neural Probabilistic Language Model

Long Short-Term Memory

Acceleration of stochastic approximation by averaging

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Improving Language Understanding by Generative Pre-Training

Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies

Field of Study

Venue Information

Name

Type

URL

Alternate Names