How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions (2019-02-01T00:00:00.000000Z)

TL;DR

It is empirically demonstrate that the performance of CLE models largely depends on the task at hand and that optimizing CLE models for BLI may hurt downstream performance, and indicates the most robust supervised and unsupervised CLE models.

Abstract

Cross-lingual word embeddings (CLEs) facilitate cross-lingual transfer of NLP models. Despite their ubiquitous downstream usage, increasingly popular projection-based CLE models are almost exclusively evaluated on bilingual lexicon induction (BLI). Even the BLI evaluations vary greatly, hindering our ability to correctly interpret performance and properties of different CLE models. In this work, we take the first step towards a comprehensive evaluation of CLE models: we thoroughly evaluate both supervised and unsupervised CLE models, for a large number of language pairs, on BLI and three downstream tasks, providing new insights concerning the ability of cutting-edge CLE models to support cross-lingual NLP. We empirically demonstrate that the performance of CLE models largely depends on the task at hand and that optimizing CLE models for BLI may hurt downstream performance. We indicate the most robust supervised and unsupervised CLE models and emphasize the need to reassess simple baselines, which still display competitive performance across the board. We hope our work catalyzes further research on CLE evaluation and model analysis.

Authors

Sebastian Ruder

20 papers

Ivan Vulic

27 papers

Goran Glavas

5 papers

TL;DR

Abstract

Authors

References66 items

Cross-lingual Language Model Pretraining

Improving Unsupervised Word-by-Word Translation with Language Model and Denoising Autoencoder

Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

Unsupervised Hyperalignment for Multilingual Word Embeddings

XNLI: Evaluating Cross-lingual Sentence Representations

Unsupervised Cross-lingual Transfer of Word Embedding Spaces

Gromov-Wasserstein Alignment of Word Embedding Spaces

A Discriminative Latent-Variable Model for Bilingual Lexicon Induction

Improving Cross-Lingual Word Embeddings by Meeting in the Middle

Unsupervised Multilingual Word Embeddings

The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing

The Limitations of Cross-language Word Embeddings Evaluation

Unsupervised Alignment of Embeddings with Wasserstein Procrustes

A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings

On the Limitations of Unsupervised Bilingual Dictionary Induction

Unsupervised Cross-Lingual Information Retrieval Using Monolingual Data Only

Generalizing and Improving Bilingual Word Embedding Mappings with a Multi-Step Framework of Linear Transformations

Phrase-Based & Neural Unsupervised Machine Translation

Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion

Non-Adversarial Unsupervised Word Translation

Unsupervised Neural Machine Translation

Word Translation Without Parallel Data

Cheap Translation for Cross-Lingual Named Entity Recognition

Learning bilingual word embeddings with (almost) no bilingual data

Adversarial Training for Unsupervised Bilingual Lexicon Induction

A Survey of Cross-lingual Word Embedding Models

A survey of cross-lingual embedding models

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Bootstrapping Unsupervised Bilingual Lexicon Induction

Offline bilingual word vectors, orthogonal transformations and the inverted softmax

Enhanced LSTM for Natural Language Inference

A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments

Enriching Word Vectors with Subword Information

Gromov-Wasserstein Averaging of Kernel and Distance Matrices

Ten Pairs to Tag – Multilingual POS Tagging via Coarse Mapping between Embeddings

Cross-lingual Models of Word Embeddings: An Empirical Comparison

Massively Multilingual Word Embeddings

Bilingual Distributed Word Representations from Document-Aligned Comparable Data

A large annotated corpus for learning natural language inference

Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings

Cross-lingual Dependency Parsing Based on Distributed Representations

Hubness and Pollution: Delving into Cross-Space Mapping for Zero-Shot Learning

Bilingual Word Representations with Monolingual Quality in Mind

Inverted indexing for cross-lingual NLP

Adam: A Method for Stochastic Optimization

BilBOWA: Fast Bilingual Distributed Representations without Word Alignments

Multilingual Models for Compositional Distributed Semantics

Improving Vector Space Word Representations Using Multilingual Correlation

Bilingual Word Embeddings for Phrase-Based Machine Translation

Exploiting Similarities among Languages for Machine Translation

Inducing Crosslingual Distributed Representations of Words

More accurate tests for the statistical significance of result differences

The “independent components” of natural scenes are edge filters

A shortest augmenting path algorithm for dense and sparse linear assignment problems

A generalized solution of the orthogonal procrustes problem

Unsupervised Bilingual Lexicon Induction via Latent Variable Models

NORMA: Neighborhood Sensitive Maps for Multilingual Word Embeddings

Learning Unsupervised Word Translations Without Adversaries

GENERATIVE ADVERSARIAL NETS

(version 18

The asjp database (version

Bilingual Lexicon Induction by Learning to Combine Word-Level and Character-Level Representations

On the Role of Seed Lexicons in Learning Bilingual Word Embeddings

Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation

Translation Invariant Word Embeddings

Appendix: BLI performance (MAP) for the second batch (14) of language pairs

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names