Semi-supervised Sequence Learning

Published in

Neural Information Processing Systems(2015)

External Links:

Generate Graph

TL;DR

Two approaches to use unlabeled data to improve Sequence Learning with recurrent networks are presented and it is found that long short term memory recurrent networks after pretrained with the two approaches become more stable to train and generalize better.

Abstract

We present two approaches to use unlabeled data to improve Sequence Learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a language model in NLP. The second approach is to use a sequence autoencoder, which reads the input sequence into a vector and predicts the input sequence again. These two algorithms can be used as a "pretraining" algorithm for a later supervised sequence learning algorithm. In other words, the parameters obtained from the pretraining step can then be used as a starting point for other supervised training models. In our experiments, we find that long short term memory recurrent networks after pretrained with the two approaches become more stable to train and generalize better. With pretraining, we were able to achieve strong performance in many classification tasks, such as text classification with IMDB, DBpedia or image recognition in CIFAR-10.

Authors

Quoc V. Le

42 papers

Andrew M. Dai

8 papers

References41 items

Character-level Convolutional Networks for Text Classification

Skip-Thought Vectors

A Neural Conversational Model

Beyond short snippets: Deep networks for video classification

LSTM: A Search Space Odyssey

Semi-supervised Sequence Learning

Published in

Neural Information Processing Systems(2015)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Quoc V. Le

42 papers

Andrew M. Dai

8 papers

References41 items

Character-level Convolutional Networks for Text Classification

Skip-Thought Vectors

A Neural Conversational Model

Beyond short snippets: Deep networks for video classification

LSTM: A Search Space Odyssey

Neural Responding Machine for Short-Text Conversation

Unsupervised Learning of Video Representations using LSTMs

Grammar as a Foreign Language

On Using Very Large Target Vocabulary for Neural Machine Translation

End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results

Effective Use of Word Order for Text Categorization with Convolutional Neural Networks

Show and tell: A neural image caption generator

Addressing the Rare Word Problem in Neural Machine Translation

Sequence to Sequence Learning with Neural Networks

Recurrent Neural Network Regularization

Convolutional Neural Networks for Sentence Classification

Distributed Representations of Sentences and Documents

Stochastic Ratio Matching of RBMs for Sparse High-Dimensional Inputs

Hidden factors and hidden topics: understanding rating dimensions with review text

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Generating Sequences With Recurrent Neural Networks

ImageNet classification with deep convolutional neural networks

Semantic Compositionality through Recursive Matrix-Vector Spaces

Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Learning Word Vectors for Sentiment Analysis

A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data

Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales

A Neural Probabilistic Language Model

Learning to Forget: Continual Prediction with LSTM

Long Short-Term Memory

NewsWeeder: Learning to Filter Netnews

Learning representations by back-propagating errors

DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia

Datasets for single-label text categorization

Listen, attend and spell

Learning Algorithms for the Classification Restricted Boltzmann Machine

Convolutional Deep Belief Networks on CIFAR-10

Recurrent neural network based language model

Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies

Learning representations by back-propagation errors, nature

Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS