Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing (2019-09-15T00:00:00.000000Z)

TL;DR

This paper proposes Cross-Lingual BERT Transformation (CLBT), a simple and efficient approach to generate cross-lingual contextualized word embeddings based on publicly available pre-trained BERT models (Devlin et al., 2018).

Abstract

This paper investigates the problem of learning cross-lingual representations in a contextual space. We propose Cross-Lingual BERT Transformation (CLBT), a simple and efficient approach to generate cross-lingual contextualized word embeddings based on publicly available pre-trained BERT models (Devlin et al., 2018). In this approach, a linear transformation is learned from contextual word alignments to align the contextualized embeddings independently trained in different languages. We demonstrate the effectiveness of this approach on zero-shot cross-lingual transfer parsing. Experiments show that our embeddings substantially outperform the previous state-of-the-art that uses static embeddings. We further compare our approach with XLM (Lample and Conneau, 2019), a recently proposed cross-lingual language model trained with massive parallel data, and achieve highly competitive results.

Authors

Ting Liu

20 papers

Yijia Liu

4 papers

Wanxiang Che

19 papers

TL;DR

Abstract

Authors

References35 items

How Multilingual is Multilingual BERT?

Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

Polyglot Contextual Representations Improve Crosslingual Transfer

Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

Cross-lingual Language Model Pretraining

Near or Far, Wide Range Zero-Shot Cross-Lingual Dependency Parsing

Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog

Deep contextualized word representations for detecting sarcasm and irony

Gromov-Wasserstein Alignment of Word Embedding Spaces

Unsupervised Multilingual Word Embeddings

Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation

Deep Contextualized Word Representations

Word Translation Without Parallel Data

Learned in Translation: Contextualized Word Vectors

Semi-supervised sequence tagging with bidirectional language models

Universal Dependencies 2.1

Offline bilingual word vectors, orthogonal transformations and the inverted softmax

Deep Biaffine Attention for Neural Dependency Parsing

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Enriching Word Vectors with Subword Information

A Representation Learning Framework for Multi-Source Transfer Parsing

Cross-lingual Dependency Parsing Based on Distributed Representations

Hubness and Pollution: Delving into Cross-Space Mapping for Zero-Shot Learning

Adam: A Method for Stochastic Optimization

Distributed Word Representation Learning for Cross-Lingual Dependency Parsing

Learning Bilingual Word Representations by Marginalizing Alignments

Exploiting Similarities among Languages for Machine Translation

Universal Dependency Annotation for Multilingual Parsing

Inducing Crosslingual Distributed Representations of Words

Vine Pruning for Efficient Multi-Pass Dependency Parsing

cdec: A Decoder, Alignment, and Learning Framework for Finite- State and Context-Free Translation Models

SENSEVAL-2: Overview

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation

Europarl: A Parallel Corpus for Statistical Machine Translation

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names