Lexical Normalization for Code-switched Data and its Effect on POS Tagging (2020-06-01T00:00:00.000000Z)

TL;DR

This paper proposes three normalization models specifically designed to handle code-switched data which are evaluated for two language pairs: Indonesian-English and Turkish-German, and introduces novel normalization layers and their corresponding language ID and POS tags for the dataset.

Abstract

Lexical normalization, the translation of non-canonical data to standard language, has shown to improve the performance of many natural language processing tasks on social media. Yet, using multiple languages in one utterance, also called code-switching (CS), is frequently overlooked by these normalization systems, despite its common use in social media. In this paper, we propose three normalization models specifically designed to handle code-switched data which we evaluate for two language pairs: Indonesian-English and Turkish-German. For the latter, we introduce novel normalization layers and their corresponding language ID and POS tags for the dataset, and evaluate the downstream effect of normalization on POS tagging. Results show that our CS-tailored normalization models significantly outperform monolingual ones, and lead to 5.4% relative performance increase for POS tagging as compared to unnormalized input.

Authors

Rob van der Goot

9 papers

Özlem Çetinoğlu

1 papers

TL;DR

Abstract

Authors

References50 items

Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP

LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation

Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection

Enhancing BERT for Lexical Normalization

Normalization of Indonesian-English Code-Mixed Twitter Data

Normalising Non-standardised Orthography in Algerian Code-switched User-generated Data

MoNoise: A Multi-lingual and Easy-to-use Lexical Normalization Tool

Adapting Sequence to Sequence models for Text Normalization in Social Media

A Fast, Compact, Accurate Model for Language Identification of Codemixed Text

Joint Part-of-Speech and Language ID Tagging for Code-Switched Data

Universal Dependency Parsing for Hindi-English Code-Switching

Word Translation Without Parallel Data

To normalize, or not to normalize: The impact of normalization on Part-of-Speech tagging

Parser Adaptation for Social Media by Integrating Normalization

Joining Hands: Exploiting Monolingual Treebanks for Parsing of Code-mixing Data

Universal Dependencies for Turkish

Part of Speech Tagging for Code Switched Data

Part of Speech Annotation of a Turkish-German Code-Switching Corpus

Bag of Tricks for Efficient Text Classification

A Turkish-German Code-Switching Corpus

Universal Dependencies v1: A Multilingual Treebank Collection

Multilingual Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Models and Auxiliary Loss

Shallow Parsing Pipeline - Hindi-English Code-Mixed Social Media Text

Overview of FIRE-2015 Shared Task on Mixed Script Information Retrieval

Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition

NCSU-SAS-Ning: Candidate Generation and Feature Engineering for Supervised Lexical Normalization

Identifying Languages at the Word Level in Code-Mixed Indian Social Media Text

Overview for the Second Shared Task on Language Identification in Code-Switched Data

Efficient Higher-Order CRFs for Morphological Tagging

Twitter Part-of-Speech Tagging for All: Overcoming Sparse and Noisy Data

Universal Dependency Annotation for Multilingual Parsing

Polyglot: Distributed Word Representations for Multilingual NLP

What to do about bad language on the internet

A Character-Level Machine Translation Approach for Normalization of SMS Abbreviations

Lexical Normalisation of Short Text Messages: Makn Sens a #twitter

TweetMotif: Exploratory Search and Topic Summarization for Twitter

A Phrase-Based Statistical Model for SMS Text Normalization

A Systematic Comparison of Various Statistical Alignment Models

Social Motivations For Codeswitching: Evidence from Africa

Norm It! Lexical Normalization for Italian and Its Downstream Effects for Dependency Parsing

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Lexical normalization of roman Urdu text

Normalising Slovene data: historical texts vs. user-generated content

Improving the utility of social media with Natural Language Processing

Introducción a la Tarea Compartida Tweet-Norm 2013: Normalización Léxica de Tuits en Español

Random Forests

Universal dependencies 2.5. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL)

Petya Osenova, RobertÖstling, Lilja Øvrelid, Niko Partanen, Elena Pascual, Marco Passarotti, Agnieszka Patejuk, Guilherme Paulino-Passos, Angelika Peljak-Łapińska, Siyao Peng

Discourse strategies, volume 1

Adaptive parsercentric text normalization

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names