CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain (2021-12-16T00:00:00.000000Z)

TL;DR

The results highlight the importance of specialized language models, such as CLIN-X, for concept extraction in non-standard domains, but also show that the task-agnostic model architecture is robust across the tested tasks and languages so that domain- or task-specific adaptations are not required.

Abstract

MOTIVATION The field of natural language processing (NLP) has recently seen a large change towards using pre-trained language models for solving almost any task. Despite showing great improvements in benchmark datasets for various tasks, these models often perform sub-optimal in non-standard domains like the clinical domain where a large gap between pre-training documents and target documents is observed. In this paper, we aim at closing this gap with domain-specific training of the language model and we investigate its effect on a diverse set of downstream tasks and settings. RESULTS We introduce the pre-trained CLIN-X (Clinical XLM-R) language models and show how CLIN-X outperforms other pre-trained transformer models by a large margin for ten clinical concept extraction tasks from two languages. In addition, we demonstrate how the transformer model can be further improved with our proposed task- and language-agnostic model architecture based on ensembles over random splits and cross-sentence context. Our studies in low-resource and transfer settings reveal stable model performance despite a lack of annotated data with improvements of up to 47 F1 points when only 250 labeled sentences are available. Our results highlight the importance of specialized language models, such as CLIN-X, for concept extraction in non-standard domains, but also show that our task-agnostic model architecture is robust across the tested tasks and languages so that domain- or task-specific adaptations are not required. AVAILABILITY The CLIN-X language models and source code for fine-tuning and transferring the model are publicly available at https://github.com/boschresearch/clin_x/ and the huggingface model hub.

Authors

References44 items

Boosting Transformers for Job Expression Extraction and Classification in a Low-Resource Setting

NLP applied to occupational health: MEDDOPROF shared task at IberLEF 2021 on automatic recognition, classification and normalization of professions and occupations from medical texts

To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning

FLERT: Document-Level Features for Named Entity Recognition

NLNDE at CANTEMIST: Neural Sequence Labeling and Parsing Approaches for Clinical Concept Extraction

BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition

HunFlair: an easy-to-use tool for state-of-the-art biomedical named entity recognition

NLNDE: The Neither-Language-Nor-Domain-Experts' Way of Spanish Medical Document De-Identification

The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain

Exploring Cross-sentence Contexts for Named Entity Recognition with BERT

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

MT-Clinical BERT: Scaling Clinical Information Extraction with Multitask Learning

MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers

Unsupervised Cross-lingual Representation Learning at Scale

PharmaCoNER: Pharmacological Substances, Compounds and proteins Named Entity Recognition track

Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases

We Need to Talk about Standard Splits

Robust Representation Learning of Biomedical Names

Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

Publicly Available Clinical BERT Embeddings

SciBERT: A Pretrained Language Model for Scientific Text

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Few-shot Learning for Named Entity Recognition in Medical Text

Contextual String Embeddings for Sequence Labeling

Deep Contextualized Word Representations

Transfer Learning for Named-Entity Recognition with Neural Networks

Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task Track 1

Challenges in clinical natural language processing for automated disorder normalization

Evaluating temporal relations in clinical text: 2012 i2b2 Challenge

2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text

Natural Language Processing (Almost) from Scratch

Evaluating the state-of-the-art in automatic de-identification.

Optimal search strategies for retrieving scientifically strong studies of treatment from Medline: analytical survey

Exploiting Context for Biomedical Entity Recognition: From Syntax to the Web

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

How Do Your Biomedical Named Entity Models Generalize to Novel Entities?

HUNER: improving biomedical NER with pretraining

Named Entity Recognition, Concept Normalization and Clinical Coding: Overview of the Cantemist Track for Cancer Text Mining in Spanish, Corpus, Guidelines, Methods and Results

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Automatic De-identification of Medical Texts in Spanish: the MEDDOCAN Track, Corpus, Guidelines, Methods and Evaluation of Results

Clinical information extraction applications: A literature review

The MeSpEN Resource for English-Spanish Medical Machine Translation and Terminologies : Census of Parallel Corpora , Glossaries and Term Translations

Committee on Computational Linguistics

classiﬁcation and normalization of professions and occupations from medical texts