PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining

Published in

North American Chapter of the Association for C...(2021)

External Links:

Generate Graph

TL;DR

PARADISE (PARAllel & Denoising Integration in SEquence-to-sequence models), which extends the conventional denoising objective used to train these models by replacing words in the noised sequence according to a multilingual dictionary, and predicting the reference translationaccording to a parallel corpus instead of recovering the original sequence.

Abstract

Despite the success of multilingual sequence-to-sequence pretraining, most existing approaches rely on monolingual corpora and do not make use of the strong cross-lingual signal contained in parallel data. In this paper, we present PARADISE (PARAllel &Denoising Integration in SEquence-to-sequence models), which extends the conventional denoising objective used to train these models by (i) replacing words in the noised sequence according to a multilingual dictionary, and (ii) predicting the reference translation according to a parallel corpus instead of recovering the original sequence. Our experiments on machine translation and cross-lingual natural language inference show an average improvement of 2.0 BLEU points and 6.7 accuracy points from integrating parallel data into pretraining, respectively, obtaining results that are competitive with several popular models at a fraction of their computational cost.

Authors

Mikel Artetxe

10 papers

Machel Reid

3 papers

References48 items

nmT5 - Is parallel data still relevant for pre-training massively multilingual language models?

mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

Identifying Elements Essential for BERT’s Multilinguality

DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining

Published in

North American Chapter of the Association for C...(2021)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Mikel Artetxe

10 papers

Machel Reid

3 papers

References48 items

nmT5 - Is parallel data still relevant for pre-training massively multilingual language models?

mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

Identifying Elements Essential for BERT’s Multilinguality

DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

Explicit Alignment Objectives for Multilingual Bidirectional Encoders

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages

A Call for More Rigor in Unsupervised Cross-lingual Learning

XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

Multilingual Denoising Pre-training for Neural Machine Translation

A Massive Collection of Cross-Lingual Web-Document Pairs

Unsupervised Cross-lingual Representation Learning at Scale

Emerging Cross-lingual Structure in Pretrained Language Models

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation

PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification

How Multilingual is Multilingual BERT?

MASS: Masked Sequence to Sequence Pre-training for Language Generation

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Polyglot Contextual Representations Improve Crosslingual Transfer

The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English

Cross-lingual Language Model Pretraining

XNLI: Evaluating Cross-lingual Sentence Representations

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

A Call for Clarity in Reporting BLEU Scores

Word Translation Without Parallel Data

Mixed Precision Training

The IIT Bombay English-Hindi Parallel Corpus

Findings of the 2017 Conference on Machine Translation (WMT17)

Findings of the 2016 Conference on Machine Translation

Edinburgh Neural Machine Translation Systems for WMT 16

OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles

The United Nations Parallel Corpus v1.0

Findings of the 2009 Workshop on Statistical Machine Translation

Bleu: a Method for Automatic Evaluation of Machine Translation

2020) results included) in Table 7. We note that on XNLI that mBART underperforms XLM-R-large, however that may be attributed to the fact that XLM-R

D Additional results We list detailed results by language in this section with results on XNLI in Table 8, PAWS-X in Table 6, and our machine translation ablation

ELECTRA: Pretraining Text Encoders as Discriminators Rather Than Generators

D Machine translation finetuning We finetune our models using the same setup as mBART, warming up the learning rate to 3× 10−5 over 2500 iterations and then decaying with a poly

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

ParaCrawl: Web-scale parallel corpora for the languages of the EU

periments are performed with fairseq (Ott et al., 2019)

Overview of the IWSLT 2017 Evaluation Campaign

Karin Verspoor, and Marcos Zampieri

The IWSLT 2015 Evaluation Campaign

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

North American Chapter of the Association for Computational Linguistics

Type

conference

URL

https://www.aclweb.org/portal/naacl

Alternate Names

North Am Chapter Assoc Comput Linguistics
NAACL