Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting

Published in

Conference on Empirical Methods in Natural Lang...(2021)

External Links:

Generate Graph DownloadPDF

TL;DR

This paper conducts extensive experiments by using SSR to improve the typical Seq2Seq pre-trained model T5 in a continual pre-training setting and show substantial improvements over T5 on various natural language generation tasks.

Abstract

In this paper, we propose Sequence Span Rewriting (SSR), a self-supervised task for sequence-to-sequence (Seq2Seq) pre-training. SSR learns to refine the machine-generated imperfect text spans into ground truth text. SSR provides more fine-grained and informative supervision in addition to the original text-infilling objective. Compared to the prevalent text infilling objectives for Seq2Seq pre-training, SSR is naturally more consistent with many downstream generation tasks that require sentence rewriting (e.g., text summarization, question generation, grammatical error correction, and paraphrase generation). We conduct extensive experiments by using SSR to improve the typical Seq2Seq pre-trained model T5 in a continual pre-training setting and show substantial improvements over T5 on various natural language generation tasks.

Authors

Furu Wei

37 papers

Tao Ge

6 papers

Wangchunshu Zhou

4 papers

References67 items

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

Pre-training Text-to-Text Transformers for Concept-centric Common Sense

Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction

Pre-training via Paraphrasing

BERT Loses Patience: Fast and Robust Inference with Early Exit

Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting

Published in

Conference on Empirical Methods in Natural Lang...(2021)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Furu Wei

37 papers

Tao Ge

6 papers

Wangchunshu Zhou

4 papers

References67 items

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

Pre-training Text-to-Text Transformers for Concept-centric Common Sense

Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction

Pre-training via Paraphrasing

BERT Loses Patience: Fast and Robust Inference with Early Exit

Language Models are Few-Shot Learners

The Right Tool for the Job: Matching Model and Instance Complexities

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning

BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

Pseudo-Bidirectional Decoding for Local Sequence Transduction

Improving Grammatical Error Correction with Machine Translation Pairs

Understanding Knowledge Distillation in Non-autoregressive Machine Translation

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

An Empirical Study of Incorporating Pseudo Data into Grammatical Error Correction

Denoising based Sequence-to-Sequence Pre-training for Text Generation

Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data

The BEA-2019 Shared Task on Grammatical Error Correction

Text Summarization with Pretrained Encoders

RoBERTa: A Robustly Optimized BERT Pretraining Approach

SpanBERT: Improving Pre-training by Representing and Predicting Spans

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Energy and Policy Considerations for Deep Learning in NLP

Defending Against Neural Fake News

Unified Language Model Pre-training for Natural Language Understanding and Generation

MASS: Masked Sequence to Sequence Pre-training for Language Generation

Are Sixteen Heads Really Better than One?

The Curious Case of Neural Text Degeneration

Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization

Born Again Neural Networks

Harvesting Paragraph-level Question-Answer Pairs from Wikipedia

Deep Contextualized Word Representations

Attention is All you Need

Get To The Point: Summarization with Pointer-Generator Networks

Toward Controlled Generation of Text

Sequence-Level Knowledge Distillation

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Deep Residual Learning for Image Recognition

Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Teaching Machines to Read and Comprehend

Distilling the Knowledge in a Neural Network

FitNets: Hints for Thin Deep Nets

CIDEr: Consensus-based image description evaluation

The CoNLL-2014 Shared Task on Grammatical Error Correction

Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English

Mining Revision Log of Language Learning SNS for Automated Japanese Error Correction of Second Language Learners

A New Dataset and Method for Automatically Grading ESOL Texts

Curriculum learning

METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

ROUGE: A Package for Automatic Evaluation of Summaries

Bleu: a Method for Automatic Evaluation of Machine Translation

Meta Learning for Knowledge Distillation

ELECTRA: pre-training text encoders as discriminators rather than generators

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Annual Conference on Neural Information Processing Systems 2019

Improving Language Understanding by Generative Pre-Training

Active Learning Literature Survey

The Boosting Approach to Machine Learning An Overview

The computer learner corpus: a versatile new source of data for SLA research

Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 1236–1242, Hong Kong, China

Proceedings of Machine Learning Research

Field of Study

Computer Science

Venue Information

Name

Conference on Empirical Methods in Natural Language Processing

Type

conference

URL

https://www.aclweb.org/portal/emnlp

Alternate Names

Empir Method Nat Lang Process
Empirical Methods in Natural Language Processing
Conf Empir Method Nat Lang Process
EMNLP