Subword Mapping and Anchoring across Languages (2021-09-09T00:00:00.000000Z)

TL;DR

The benefits of SMALA for cross-lingual natural language inference (XNLI), where it improves zero-shot transfer to an unseen language without task-specific data, but only by sharing subword embeddings, and in neural machine translation, it shows that joint subword vocabularies obtained with SMAL a lead to higher BLEU scores on sentences that contain many false positives and false negatives.

Abstract

State-of-the-art multilingual systems rely on shared vocabularies that sufficiently cover all considered languages. To this end, a simple and frequently used approach makes use of subword vocabularies constructed jointly over several languages. We hypothesize that such vocabularies are suboptimal due to false positives (identical subwords with different meanings across languages) and false negatives (different subwords with similar meanings). To address these issues, we propose Subword Mapping and Anchoring across Languages (SMALA), a method to construct bilingual subword vocabularies. SMALA extracts subword alignments using an unsupervised state-of-the-art mapping technique and uses them to create cross-lingual anchors based on subword similarities. We demonstrate the benefits of SMALA for cross-lingual natural language inference (XNLI), where it improves zero-shot transfer to an unseen language without task-specific data, but only by sharing subword embeddings. Moreover, in neural machine translation, we show that joint subword vocabularies obtained with SMALA lead to higher BLEU scores on sentences that contain many false positives and false negatives.

Authors

Giorgos Vernikos

1 papers

Andrei Popescu-Belis

1 papers

TL;DR

Abstract

Authors

References62 items

Multi-view Subword Regularization

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

Beyond Offline Mapping: Learning Cross-lingual Word Embeddings through Context Anchoring

UNKs Everywhere: Adapting Multilingual Language Models to New Scripts

As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages

When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models

Probing Pretrained Language Models for Lexical Semantics

On Romanization for Model Transfer Between Scripts in Neural Machine Translation

Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT

Neural Machine Translation

SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings

Extending Multilingual BERT to Low-Resource Languages

Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks

XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

Multilingual Denoising Pre-training for Neural Machine Translation

Unsupervised Cross-lingual Representation Learning at Scale

Emerging Cross-lingual Structure in Pretrained Language Models

BPE-Dropout: Simple and Effective Subword Regularization

On the Cross-lingual Transferability of Monolingual Representations

Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework

From English To Foreign Languages: Transferring Pre-trained Language Models

Bilingual Lexicon Induction with Semi-supervision in Non-Isometric Embedding Spaces

How Multilingual is Multilingual BERT?

Analyzing the Limitations of Cross-lingual Word Embedding Mappings

Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

Massively Multilingual Neural Machine Translation

Cross-lingual Language Model Pretraining

Integrating Weakly Supervised Word Sense Disambiguation into Neural Machine Translation

XNLI: Evaluating Cross-lingual Sentence Representations

Transfer

A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings

On the Limitations of Unsupervised Bilingual Dictionary Induction

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

A Call for Clarity in Reporting BLEU Scores

Phrase-Based & Neural Unsupervised Machine Translation

Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion

Word Translation Without Parallel Data

Improving Word Sense Disambiguation in Neural Machine Translation with Sense Embeddings

Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

Learning bilingual word embeddings with (almost) no bilingual data

Attention is All you Need

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

OpenNMT: Open-Source Toolkit for Neural Machine Translation

Enriching Word Vectors with Subword Information

Learning Crosslingual Word Embeddings without Bilingual Corpora

The United Nations Parallel Corpus v1.0

From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

Neural Machine Translation of Rare Words with Subword Units

Bilingual Word Representations with Monolingual Quality in Mind

Adam: A Method for Stochastic Optimization

Exploiting Similarities among Languages for Machine Translation

A Simple, Fast, and Effective Reparameterization of IBM Model 2

Efficient Estimation of Word Representations in Vector Space

Japanese and Korean voice search

Moses: Open Source Toolkit for Statistical Machine Translation

Good as new

Bleu: a Method for Automatic Evaluation of Machine Translation

Do explicit alignments robustly improve multilingual encoders

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation

We demonstrate experimentally the beneﬁts of SMALA for cross-language natural language inference (XNLI) (Section 5)

Field of Study

Venue Information

Name

Type

URL

Alternate Names