Towards Robust Named Entity Recognition for Historic German

Published in

(2019)

External Links:

Generate Graph DownloadPDF

TL;DR

Results show that using SMLM can achieve comparable results for Historic named entity recognition, even when they are only trained on contemporary texts, and it is shown that language model pre-training can be more effective than using transfer-learning with labeled datasets.

Abstract

In this paper we study the influence of using language model pre-training for named entity recognition for Historic German. We achieve new state-of-the-art results using carefully chosen training data for language models. For a low-resource domain like named entity recognition for Historic German, language model pre-training can be a strong competitor to CRF-only methods. We show that language model pre-training can be more effective than using transfer-learning with labeled datasets. Furthermore, we introduce a new language model pre-training objective, synthetic masked language model pre-training (SMLM), that allows a transfer from one domain (contemporary texts) to another domain (historical texts) by using only the same (character) vocabulary. Results show that using SMLM can achieve comparable results for Historic named entity recognition, even when they are only trained on contemporary texts. Our pre-trained character-based language models improve upon classical CRF-based methods and previous work on Bi-LSTMs by boosting F1 score performance by up to 6%.

Authors

Stefan Schweter

2 papers

Johannes Baiter

1 papers

References20 items

Dissecting Contextual Word Embeddings: Architecture and Representation

Contextual String Embeddings for Sequence Labeling

A Named Entity Recognition Shootout for German

Deep Contextualized Word Representations

Low-Resource Named Entity Recognition with Cross-lingual, Character-Level Neural Conditional Random Fields

Towards Robust Named Entity Recognition for Historic German

Published in

(2019)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Stefan Schweter

2 papers

Johannes Baiter

1 papers

References20 items

Dissecting Contextual Word Embeddings: Architecture and Representation

Contextual String Embeddings for Sequence Labeling

A Named Entity Recognition Shootout for German

Deep Contextualized Word Representations

Low-Resource Named Entity Recognition with Cross-lingual, Character-Level Neural Conditional Random Fields

Multi-channel BiLSTM-CRF Model for Emerging Named Entity Recognition in Social Media

Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

Semi-supervised sequence tagging with bidirectional language models

An Open Corpus for Named Entity Recognition in Historic Newspapers

Neural Architectures for Named Entity Recognition

End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

Bidirectional LSTM-CRF Models for Sequence Tagging

GloVe: Global Vectors for Word Representation

NoSta-D Named Entity Annotation for German: Guidelines and Dataset

Distributed Representations of Words and Phrases and their Compositionality

Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages

Design Challenges and Misconceptions in Named Entity Recognition

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

2018) for the feature-based approach does not work well. Thus, we perform a per-layer analysis that trains a model with a specific layer from the multi-lingual BERT model

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687