Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019-10-23T00:00:00.000000Z)

TL;DR

This systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks and achieves state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.

Abstract

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.

Authors

Noam Shazeer

7 papers

Peter J. Liu

7 papers

Sharan Narang

8 papers

TL;DR

Abstract

Authors

References134 items

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

FreeLB: Enhanced Adversarial Training for Language Understanding

TinyBERT: Distilling BERT for Natural Language Understanding

Simple, Scalable Adaptation for Neural Machine Translation

CTRL: A Conditional Transformer Language Model for Controllable Generation

The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

A Hybrid Neural Network Model for Commonsense Reasoning

RoBERTa: A Robustly Optimized BERT Pretraining Approach

The CommitmentBank: Investigating projection in naturally occurring discourse

SpanBERT: Improving Pre-training by Representing and Predicting Spans

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Neural transfer learning for natural language processing

Transfer Learning in Natural Language Processing

Defending Against Neural Fake News

A Surprisingly Robust Trick for the Winograd Schema Challenge

Unified Language Model Pre-training for Natural Language Understanding and Generation

MASS: Masked Sequence to Sequence Pre-training for Language Generation

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Unifying Question Answering and Text Classification via Span Extraction

Fine-tune BERT for Extractive Summarization

Cloze-driven Pretraining of Self-attention Networks

To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

SciBERT: A Pretrained Language Model for Scientific Text

Parameter-Efficient Transfer Learning for NLP

Multi-Task Deep Neural Networks for Natural Language Understanding

Memory-Efficient Adaptive Optimization for Large-Scale Learning

Cross-lingual Language Model Pretraining

Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling

Rethinking ImageNet Pre-Training

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Measuring the Effects of Data Parallelism on Neural Network Training

Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

Mesh-TensorFlow: Deep Learning for Supercomputers

ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension

Music Transformer: Generating Music with Long-Term Structure

WiC: 10, 000 Example Pairs for Evaluating Context-Sensitive Representations

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Character-Level Language Modeling with Deeper Self-Attention

The Natural Language Decathlon: Multitask Learning as Question Answering

Snorkel MeTaL: Weak Supervision for Multi-Task Learning

A Simple Method for Commonsense Reasoning

Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences

Neural Network Acceptability Judgments

Do Better ImageNet Models Transfer Better?

Exploring the Limits of Weakly Supervised Pretraining

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

A Call for Clarity in Reporting BLEU Scores

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Adafactor: Adaptive Learning Rates with Sublinear Memory Cost

Self-Attention with Relative Position Representations

SentEval: An Evaluation Toolkit for Universal Sentence Representations

Learning Word Vectors for 157 Languages

Deep Contextualized Word Representations

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

An efficient framework for learning sentence representations

Generating Wikipedia by Summarizing Long Sequences

Universal Language Model Fine-tuning for Text Classification

Deep Learning Scaling is Predictable, Empirically

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

An Overview of Multi-Task Learning in Deep Neural Networks

Attention is All you Need

A Deep Reinforced Model for Abstractive Summarization

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Get To The Point: Summarization with Pointer-Generator Networks

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

NewsQA: A Machine Comprehension Dataset

Unsupervised Pretraining for Sequence to Sequence Learning

Federated Learning: Strategies for Improving Communication Efficiency