How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks (2017-02-07T00:00:00.000000Z)

TL;DR

It is proposed that evaluation of word representation evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning).

Abstract

Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evaluated. In addition, recent practice in word embeddings points towards importance of learning specialized representations. We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible. Specifically, we propose that evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning). In order to illustrate significance of such analysis, a comprehensive evaluation of selected word embeddings is presented. Proposed approach yields a more complete picture and brings new insight into performance characteristics, for instance information about word similarity or analogy tends to be non--linearly encoded in the embedding space, which questions the cosine-based, unsupervised, evaluation methods. All results and analysis scripts are available online.

Authors

Stanislaw Jastrzebski

5 papers

Wojciech M. Czarnecki

7 papers

Damian Lesniak

1 papers

TL;DR

Abstract

Authors

References42 items

Expanding Subjective Lexicons for Social Media Mining with Embedding Subspaces

Creating Causal Embeddings for Question Answering with Minimal Supervision

Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance

Improving Reliability of Word Similarity Evaluation by Redesigning Annotation Task and Performance Measure

Sparsifying Word Representations for Deep Unordered Sentence Modeling

Evaluating Word Embeddings Using a Representative Suite of Practical Tasks

Word Embedding Calculus in Meaningful Ultradense Subspaces

Problems With Evaluation of Word Embeddings Using Word Similarity Tasks

Neural Architectures for Named Entity Recognition

Linear Algebraic Structure of Word Senses, with Applications to Polysemy

A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification

A Primer on Neural Network Models for Natural Language Processing

Character-level Convolutional Networks for Text Classification

What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation

Evaluation of Word Vector Representations by Subspace Alignment

Evaluation methods for unsupervised word embeddings

How to Generate a Good Word Embedding

Learning Word Representations from Scarce and Noisy Data with Embedding Subspaces

Learning Word Representations by Jointly Modeling Syntagmatic and Paradigmatic Relations

"The Sum of Its Parts": Joint Learning of Word and Phrase Representations with Autoencoders

Non-distributional Word Vector Representations

Improving Distributional Similarity with Lessons Learned from Word Embeddings

Big Data Small Data, In Domain Out-of Domain, Known Word Unknown Word: The Impact of Word Representations on Sequence Labelling Tasks

Embedding Word Similarity with Neural Machine Translation

GloVe: Global Vectors for Word Representation

Convolutional Neural Networks for Sentence Classification

SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

WordRep: A Benchmark for Research on Learning Word Representations

How much do word embeddings encode about syntax?

Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors

Large-Scale Linear RankSVM

Word Embeddings through Hellinger PCA

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Better Word Representations with Recursive Neural Networks for Morphology

Representation Learning: A Review and New Perspectives

Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach

Two/Too Simple Adaptations of Word2Vec for Syntax Problems

Multimodal Distributional Semantics

The Sum of its Parts

Placing search in context: the concept revisited

of the Association for Computational Linguistics

information about the task and how can it be measured given some data representation

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names