Distributed Representations of Sentences and Documents

Published in

International Conference on Machine Learning(2014)

External Links:

Generate Graph

TL;DR

Paragraph Vector is an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents, and its construction gives the algorithm the potential to overcome the weaknesses of bag-of-words models.

Abstract

Many machine learning algorithms require the input to be represented as a fixed-length feature vector. When it comes to texts, one of the most common fixed-length features is bag-of-words. Despite their popularity, bag-of-words features have two major weaknesses: they lose the ordering of the words and they also ignore semantics of the words. For example, "powerful," "strong" and "Paris" are equally distant. In this paper, we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. Our algorithm represents each document by a dense vector which is trained to predict words in the document. Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models. Empirical results show that Paragraph Vectors outperforms bag-of-words models as well as other techniques for text representations. Finally, we achieve new state-of-the-art results on several text classification and sentiment analysis tasks.

Authors

Quoc V. Le

42 papers

Tomas Mikolov

9 papers

References42 items

Reasoning With Neural Tensor Networks for Knowledge Base Completion

DeViSE: A Deep Visual-Semantic Embedding Model

Distributed Representations of Words and Phrases and their Compositionality

Bilingual Word Embeddings for Phrase-Based Machine Translation

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Distributed Representations of Sentences and Documents

Published in

International Conference on Machine Learning(2014)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Quoc V. Le

42 papers

Tomas Mikolov

9 papers

References42 items

Reasoning With Neural Tensor Networks for Knowledge Base Completion

DeViSE: A Deep Visual-Semantic Embedding Model

Distributed Representations of Words and Phrases and their Compositionality

Bilingual Word Embeddings for Phrase-Based Machine Translation

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Exploiting Similarities among Languages for Machine Translation

Modeling Documents with Deep Boltzmann Machines

Linguistic Regularities in Continuous Space Word Representations

Combining Heterogeneous Models for Measuring Relational Similarity

Multi-Step Regression Learning for Compositional Distributional Semantics

Efficient Estimation of Word Representations in Vector Space

A Neural Autoregressive Topic Model

Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Improving Word Representations via Global Context and Multiple Word Prototypes

Training Restricted Boltzmann Machines on Word Observations

Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Compositional Matrix-Space Models for Sentiment Analysis

Parsing Natural Scenes and Natural Language with Recursive Neural Networks

Learning Word Vectors for Sentiment Analysis

Natural Language Processing (Almost) from Scratch

Composition in Distributional Models of Semantics

Estimating Linear Models for Compositional Distributional Semantics

Word Representations: A Simple and General Method for Semi-Supervised Learning

Large-scale image retrieval with compressed Fisher vectors

From Frequency to Meaning: Vector Space Models of Semantics

A Scalable Hierarchical Distributed Language Model

A unified architecture for natural language processing: deep neural networks with multitask learning

Fisher Kernels on Visual Vocabularies for Image Categorization

Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales

Accurate Unlexicalized Parsing

A neural probabilistic language model

Exploiting Generative Models in Discriminative Classifiers

Finding Structure in Time

Learning representations by back-propagating errors

Recursive deep learning for natural language processing and computer vision

Statistical Language Models Based on Neural Networks

Neural Probabilistic Language Models

Hierarchical Probabilistic Neural Network Language Model

Learning representations by back-propagation errors, nature

Distributional Structure

Every sentence in the dataset has a label which goes from very negative to very positive in the scale from 0.0 to 1.0

Field of Study

Computer Science

Venue Information

Name

International Conference on Machine Learning

Type

conference

URL

https://icml.cc/

Alternate Names

ICML
Int Conf Mach Learn