HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019-10-09T00:00:00.000000Z)

TL;DR

The \textit{Transformers} library is an open-source library that consists of carefully engineered state-of-the art Transformer architectures under a unified API and a curated collection of pretrained models made by and available for the community.

Abstract

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{this https URL}.

Authors

Yacine Jernite

2 papers

Teven Le Scao

6 papers

Patrick von Platen

3 papers

TL;DR

Abstract

Authors

References65 items

Longformer: The Long-Document Transformer

Stanza: A Python Natural Language Processing Toolkit for Many Human Languages

jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models

Reformer: The Efficient Transformer

FlauBERT: Unsupervised Language Model Pre-training for French

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Supervised Multimodal Bitransformers for Classifying Images and Text

RoBERTa: A Robustly Optimized BERT Pretraining Approach

SpanBERT: Improving Pre-training by Representing and Predicting Spans

Large-Scale Transfer Learning for Natural Language Generation

XLNet: Generalized Autoregressive Pretraining for Language Understanding

COMET: Commonsense Transformers for Automatic Knowledge Graph Construction

What Does BERT Look at? An Analysis of BERT’s Attention

FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP

Pooled Contextualized Embeddings for Named Entity Recognition

Transfer Learning in Natural Language Processing

BERT Rediscovers the Classical NLP Pipeline

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Are Sixteen Heads Really Better than One?

The Curious Case of Neural Text Degeneration

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

SciBERT: A Pretrained Language Model for Scientific Text

TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents

Cross-lingual Language Model Pretraining

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Model Cards for Model Reporting

WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Contextual String Embeddings for Sequence Labeling

Know What You Don’t Know: Unanswerable Questions for SQuAD

Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences

Gender Bias in Coreference Resolution

Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

AllenNLP: A Deep Semantic Natural Language Processing Platform

Tensor2Tensor for Neural Machine Translation

Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Deep Contextualized Word Representations

TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

Universal Language Model Fine-tuning for Text Classification

Decoupled Weight Decay Regularization

Automatic differentiation in PyTorch

Learned in Translation: Contextualized Word Vectors

Attention is All you Need

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

RACE: Large-scale ReAding Comprehension Dataset From Examinations

OpenNMT: Open-Source Toolkit for Neural Machine Translation

LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Adam: A Method for Stochastic Optimization

The Stanford CoreNLP Natural Language Processing Toolkit

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

The Winograd Schema Challenge

Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning

The Third PASCAL Recognizing Textual Entailment Challenge

NLTK: The Natural Language Toolkit

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

The Seventh PASCAL Recognizing Textual Entailment Challenge

Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment

Automatically Constructing a Corpus of Sentential Paraphrases

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names