Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (2019-04-19T00:00:00.000000Z)

TL;DR

This paper explores the broader cross-lingual potential of mBERT (multilingual) as a zero shot language transfer model on 5 NLP tasks covering a total of 39 languages from various language families: NLI, document classification, NER, POS tagging, and dependency parsing.

Abstract

Pretrained contextual representation models (Peters et al., 2018; Devlin et al., 2018) have pushed forward the state-of-the-art on many NLP tasks. A new release of BERT (Devlin, 2018) includes a model simultaneously pretrained on 104 languages with impressive performance for zero-shot cross-lingual transfer on a natural language inference task. This paper explores the broader cross-lingual potential of mBERT (multilingual) as a zero shot language transfer model on 5 NLP tasks covering a total of 39 languages from various language families: NLI, document classification, NER, POS tagging, and dependency parsing. We compare mBERT with the best-published methods for zero-shot cross-lingual transfer and find mBERT competitive on each task. Additionally, we investigate the most effective strategy for utilizing mBERT in this manner, determine to what extent mBERT generalizes away from language specific features, and measure factors that influence cross-lingual transfer.

Authors

Mark Dredze

7 papers

Shijie Wu

4 papers

TL;DR

Abstract

Authors

References46 items

How Multilingual is Multilingual BERT?

Polyglot Contextual Representations Improve Crosslingual Transfer

Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

Cross-lingual Language Model Pretraining

Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing

Near or Far, Wide Range Zero-Shot Cross-Lingual Dependency Parsing

XNLI: Evaluating Cross-lingual Sentence Representations

Neural Cross-Lingual Named Entity Recognition with Minimal Resources

A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings

A Corpus for Multilingual Document Classification in Eight Languages

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Deep Contextualized Word Representations

The WiLI benchmark dataset for written language identification

Universal Language Model Fine-tuning for Text Classification

Word Translation Without Parallel Data

Cross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources

A Survey of Cross-lingual Word Embedding Models

Attention is All you Need

Offline bilingual word vectors, orthogonal transformations and the inverted softmax

Deep Biaffine Attention for Neural Dependency Parsing

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Layer Normalization

Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units

Gaussian Error Linear Units (GELUs)

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Massively Multilingual Word Embeddings

Universal Dependencies 1.4

Adam: A Method for Stochastic Optimization

How transferable are features in deep neural networks?

GloVe: Global Vectors for Word Representation

Distributed Representations of Words and Phrases and their Compositionality

Efficient Estimation of Word Representations in Vector Space

A Survey on Transfer Learning

The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4

Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition

Long Short-Term Memory

“Cloze Procedure”: A New Tool for Measuring Readability

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Improving Language Understanding by Generative Pre-Training

Universal dependencies 2.2. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguis

Multilingual bert readme document

Python 3 text processing with NLTK 3 cookbook : over 80 practical recipes on natural language processing techniques using Python's NLTK 3.0

Cross-Language Parser Adaptation between Related Languages

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names