User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization (2021-04-08T00:00:00.000000Z)

TL;DR

A publicly available Japanese UGT corpus that comprises 929 sentences annotated with morphological and normalization information, along with category information the authors classified for frequent UGT-specific phenomena is constructed.

Abstract

Morphological analysis (MA) and lexical normalization (LN) are both important tasks for Japanese user-generated text (UGT). To evaluate and compare different MA/LN systems, we have constructed a publicly available Japanese UGT corpus. Our corpus comprises 929 sentences annotated with morphological and normalization information, along with category information we classified for frequent UGT-specific phenomena. Experiments on the corpus demonstrated the low performance of existing MA/LN methods for non-general words and non-standard forms, indicating that the corpus would be a challenging benchmark for further research on UGT.

Authors

S. Higashiyama

1 papers

M. Utiyama

1 papers

Taro Watanabe

3 papers

User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization

TL;DR

Abstract

Authors

References29 items

Synthetic Data for English Lexical Normalization: How Close Can We Get to Manually Annotated Data?

Design and Structure of The Juman++ Morphological Analyzer Toolkit

MoNoise: A Multi-lingual and Easy-to-use Lexical Normalization Tool

A Taxonomy for In-depth Evaluation of Normalization for User Generated Content

Automatically Extracting Variant-Normalization Pairs for Japanese Text Normalization

A Transition-based Model for Joint Segmentation, POS-tagging and Normalization

Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition

NCSU-SAS-Ning: Candidate Generation and Feature Engineering for Supervised Lexical Normalization

Accurate Word Segmentation and POS Tagging for Japanese Microblogs: Corpus Annotation and Joint Modeling with Lexical Normalization

Morphological Analysis for Japanese Noisy Text based on Character-level and Word-level Normalization

Balanced corpus of contemporary written Japanese

A Log-Linear Model for Unsupervised Text Normalization

A Simple Approach to Unknown Word Processing in Japanese Morphological Analysis

Chinese Informal Word Normalization: an Experimental Study

Lexical Normalisation of Short Text Messages: Makn Sens a #twitter

Pointwise Prediction for Robust, Adaptable Japanese Morphological Analysis

Mining and Modeling Relations between Formal and Informal Chinese Phrases from Web Corpora

Applying Conditional Random Fields to Japanese Morphological Analysis

Nihongo Twitter bunsho wo taish¯o to shita k¯eretsu labeling ni yoru hy¯oki s¯ekika (Text normalization by sequence labeling for Japanese Twitter documents)

Gendai nihongo kakikotoba kinkō corpus kētairon kitēshū dai 4 ban ge (Regulations of morphological information for balanced corpus of contemporary

Construction of a blog corpus with syntactic, anaphoric, and sentiment annotations [in Japanese]

Automatic rule generation approach for morphological analysis of peculiar expressions on blog documents [in Japanese]

A multi-purpose electronic dictionary for morphological analyzers [in Japanese

Tod¯ofuken betsu zenkoku h¯ogen jiten (Dialect Dictionary of Japanese Prefectures)

Balanced Corpus of Contemporary Written Japanese

Kētaiso kaiseki no error bunseki ( Error analysis of morphological analysis )

Giongo gitaigo jiten (Phonomime and Phenomime Dictionary). Kodansha

Computational Linguistics: Technical Papers

an electronic dictionary for morphological analysis and its application to Japanese corpus linguistics [in Japanese]

Field of Study

Venue Information

Name

Type

URL

Alternate Names