A Survey of Word Embeddings Evaluation Methods (2018-01-21T00:00:00.000000Z)

TL;DR

An extensive overview of the field of word embeddings evaluation is presented, highlighting main problems and proposing a typology of approaches to evaluation, summarizing 16 intrinsic methods and 12 extrinsic methods.

Abstract

Word embeddings are real-valued word representations able to capture lexical semantics and trained on natural language corpora. Models proposing these representations have gained popularity in the recent years, but the issue of the most adequate evaluation method still remains open. This paper presents an extensive overview of the field of word embeddings evaluation, highlighting main problems and proposing a typology of approaches to evaluation, summarizing 16 intrinsic methods and 12 extrinsic methods. I describe both widely-used and experimental methods, systematize information about evaluation datasets and discuss some key challenges.

Authors

Amir Bakarov

2 papers

TL;DR

Abstract

Authors

References139 items

The Provo Corpus: A large eye-tracking corpus with predictability norms

Distributional Models of Word Meaning

Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity

An Exploration of Word Embedding Initialization in Deep-Learning Tasks

Word embeddings quantify 100 years of gender and ethnic stereotypes

Semantic Structure and Interpretability of Word Embeddings

Evaluation of word embeddings against cognitive processes: primed reaction times in lexical decision and naming tasks

Traversal-Free Word Vector Evaluation in Analogy Space

Hypothesis Testing based Intrinsic Evaluation of Word Embeddings

SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity

Automated Detection of Non-Relevant Posts on the Russian Imageboard "2ch": Importance of the Choice of Word Representations

Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading

How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks

Explaining human performance in psycholinguistic tasks with models of semantic similarity based on prediction and counting : A review and empirical validation

Automated Generation of Multilingual Clusters for the Evaluation of Distributed Representations

Story Cloze Evaluator: Vector Space Representation Evaluation by Predicting What Happens Next

An Improved Crowdsourcing Based Evaluation Technique for Word Embedding Methods

A critique of word similarity as a method for evaluating distributional semantic models

SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity

Evaluating word embeddings with fMRI and eye-tracking

Thematic fit evaluation: an aspect of selectional preferences

Evaluating Word Embeddings Using a Representative Suite of Practical Tasks

Evaluating Embeddings using Syntax-based Classification Tasks as a Proxy for Parser Performance

Capturing Discriminative Attributes in a Distributional Space: Task Proposal

Evaluating vector space models using human semantic priming results

Intrinsic Evaluations of Word Embeddings: What Can We Do Better?

Finding Non-Arbitrary Form-Meaning Systematicity Using String-Metric Learning for Kernel Regression

Evaluating embeddings on dictionary-based similarity

Evaluating multi-sense embeddings for semantic resolution monolingually and in word translation

Improving Reliability of Word Similarity Evaluation by Redesigning Annotation Task and Performance Measure

Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance

Sentence Embedding Evaluation Using Pyramid Annotation

Probing for semantic evidence of composition by means of simple classification tasks

Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings

Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t.

A comparative evaluation of off-the-shelf distributed semantic representations for modelling behavioural data

Treating Similarity with Respect: How to Evaluate Models of Meaning?

A Proposal for Linguistic Similarity Datasets Based on Commonality Lists

Problems With Evaluation of Word Embeddings Using Word Similarity Tasks

What do you know about an alligator when you know the company it keeps

Cross-lingual Models of Word Embeddings: An Empirical Comparison

Find the word that does not belong: A Framework for an Intrinsic Evaluation of Word Vector Representations

Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning

Evaluation methods for unsupervised word embeddings

Evaluation of Word Vector Representations by Subspace Alignment

Specializing Word Embeddings for Similarity or Relatedness

A large annotated corpus for learning natural language inference

Lexical Semantics and Model Theory: Together at Last?

Verb polysemy and frequency effects in thematic fit modeling

An Unsupervised Model for Instance Level Subcategorization Acquisition

SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

WordRep: A Benchmark for Research on Learning Word Representations

Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors

Linguistic Regularities in Sparse and Explicit Word Representations

Metaphor Detection with Cross-Lingual Model Transfer

How much do word embeddings encode about syntax?

Tailoring Continuous Word Representations for Dependency Parsing

A high-resolution 7-Tesla fMRI dataset from complex natural stimulation with an audio movie

Better Word Representations with Recursive Neural Networks for Morphology

Evaluating Neighbor Rank and Distance Measures as Predictors of Semantic Priming

Linguistic Regularities in Continuous Space Word Representations

The semantic priming project

Efficient Estimation of Word Representations in Vector Space

Large-scale learning of word relatedness with constraints

Abstracts From Peer-Reviewed Presentations at the Australasian Cognitive Neurosciences Conference (21st meeting of the Australasian Society for Psychophysiology), December 9-12, 2012, Macquarie University, Sydney, Australia

SemEval-2012 Task 2: Measuring Degrees of Relational Similarity

Roget's thesaurus and semantic similarity

How we BLESSed distributional semantic evaluation

Learning Word Vectors for Sentiment Analysis

A word at a time: computing word relatedness using temporal semantic analysis

Natural Language Processing (Almost) from Scratch

Distributional Memory: A General Framework for Corpus-Based Semantics

Word Representations: A Simple and General Method for Semi-Supervised Learning

Multi-Prototype Vector-Space Models of Word Meaning

From Frequency to Meaning: Vector Space Models of Semantics