Unsupervised Cross-lingual Representation Learning at Scale (2019-11-05T00:00:00.000000Z)

TL;DR

It is shown that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks, and the possibility of multilingual modeling without sacrificing per-language performance is shown for the first time.

Abstract

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code and models publicly available.

Authors

Edouard Grave

12 papers

Alexis Conneau

15 papers

Luke Zettlemoyer

46 papers

TL;DR

Abstract

Authors

References42 items

Emerging Cross-lingual Structure in Pretrained Language Models

CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

MLQA: Evaluating Cross-lingual Extractive Question Answering

Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges

How Multilingual is Multilingual BERT?

XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering

Unsupervised Data Augmentation for Consistency Training

Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

Multilingual Neural Machine Translation with Knowledge Distillation

Cross-lingual Language Model Pretraining

Adaptive Input Representations for Neural Language Modeling

XNLI: Evaluating Cross-lingual Sentence Representations

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Contextual String Embeddings for Sequence Labeling

Know What You Don’t Know: Unanswerable Questions for SQuAD

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Learning Word Vectors for 157 Languages

Deep Contextualized Word Representations

Attention is All you Need

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Efficient softmax approximation for GPUs

Bag of Tricks for Efficient Text Classification

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Neural Architectures for Named Entity Recognition

Exploring the Limits of Language Modeling

A large annotated corpus for learning natural language inference

GloVe: Global Vectors for Word Representation

Distributed Representations of Words and Phrases and their Compositionality

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Exploiting Similarities among Languages for Machine Translation

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Improving Language Understanding by Generative Pre-Training

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names