BERTweet: A pre-trained language model for English Tweets (2020-05-20T00:00:00.000000Z)

TL;DR

BERTweet is presented, the first public large-scale pre-trained language model for English Tweets, trained using the RoBERTa pre-training procedure, producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks.

Abstract

We present BERTweet, the first public large-scale pre-trained language model for English Tweets. Our BERTweet, having the same architecture as BERT-base (Devlin et al., 2019), is trained using the RoBERTa pre-training procedure (Liu et al., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base (Conneau et al., 2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks: Part-of-speech tagging, Named-entity recognition and text classification. We release BERTweet under the MIT License to facilitate future research and applications on Tweet data. Our BERTweet is available at https://github.com/VinAIResearch/BERTweet

Authors

A. Nguyen

11 papers

Dat Quoc Nguyen

12 papers

Thanh Vu

3 papers

TL;DR

Abstract

Authors

References42 items

WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets

Extracting COVID-19 Events from Twitter

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

Social media big data analytics: A survey

Unsupervised Cross-lingual Representation Learning at Scale

HuggingFace's Transformers: State-of-the-art Natural Language Processing

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Dual Adversarial Neural Transfer for Low-Resource Named Entity Recognition

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

SciBERT: A Pretrained Language Model for Scientific Text

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

SemEval-2018 Task 3: Irony Detection in English Tweets

THU_NGN at SemEval-2018 Task 3: Tweet Irony Detection with Densely connected LSTM and Multi-task Learning

Parsing Tweets into Universal Dependencies

NTUA-SLP at SemEval-2018 Task 3: Tracking Ironic Tweets using Ensembles of Word and Character Level Attentive RNNs

Decoupled Weight Decay Regularization

Results of the WNUT2017 Shared Task on Novel and Emerging Entity Recognition

Part-of-Speech Tagging for Twitter with Adversarial Neural Networks

A Multi-task Approach for Named Entity Recognition in Social Media Data

DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis

SemEval-2017 Task 4: Sentiment Analysis in Twitter

Attention is All you Need

BB_twtr at SemEval-2017 Task 4: Twitter Sentiment Analysis with CNNs and LSTMs

Bidirectional LSTM for Named Entity Recognition in Twitter Messages

Results of the WNUT16 Named Entity Recognition Shared Task

Bag of Tricks for Efficient Text Classification

End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

Neural Machine Translation of Rare Words with Subword Units

Adam: A Method for Stochastic Optimization

Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters

What to do about bad language on the internet

Automatically Constructing a Normalisation Dictionary for Microblogs

A Broad-Coverage Normalization System for Social Media Language

Named Entity Recognition in Tweets: An Experimental Study

Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments

Building a Large Annotated Corpus of English: The Penn Treebank

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Transferring from Formal Newswire Domain with Hypernet for Twitter POS Tagging

A Gold Standard Dependency Corpus for English

Lexical normalization for social media text

TYPO CORPUS

ark-tweet-nlp/downloads (twpos-data-v0.3

Field of Study

Venue Information

Name

Type

URL

Alternate Names