GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation (2022-11-18T00:00:00.000000Z)

TL;DR

GeniusAug is proposed, which first extracts the target-aware sketches from the original training set and then generates new samples based on the sketches, and is demonstrated to be a strong and ready-to-use data augmentation tool for various natural language processing (NLP) tasks.

Abstract

We introduce GENIUS: a conditional text generation model using sketches as input, which can fill in the missing contexts for a given sketch (key information consisting of textual spans, phrases, or words, concatenated by mask tokens). GENIUS is pre-trained on a large-scale textual corpus with a novel reconstruction from sketch objective using an extreme and selective masking strategy, enabling it to generate diverse and high-quality texts given sketches. Comparison with other competitive conditional language models (CLMs) reveals the superiority of GENIUS's text generation quality. We further show that GENIUS can be used as a strong and ready-to-use data augmentation tool for various natural language processing (NLP) tasks. Most existing textual data augmentation methods are either too conservative, by making small changes to the original text, or too aggressive, by creating entirely new samples. With GENIUS, we propose GeniusAug, which first extracts the target-aware sketches from the original training set and then generates new samples based on the sketches. Empirical experiments on 6 text classification datasets show that GeniusAug significantly improves the models' performance in both in-distribution (ID) and out-of-distribution (OOD) settings. We also demonstrate the effectiveness of GeniusAug on named entity recognition (NER) and machine reading comprehension (MRC) tasks. (Code and models are publicly available at https://github.com/microsoft/SCGLab and https://github.com/beyondguo/genius)

Authors

Nan Duan

17 papers

Yeyun Gong

6 papers

Yelong Shen

5 papers

TL;DR

Abstract

Authors

References68 items

GLM-130B: An Open Bilingual Pre-trained Model

Selective Text Augmentation with Word Roles for Low-Resource Text Classification

Parallel Refinements for Lexically Constrained Text Generation with BART

CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation

MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NER

FlipDA: Effective and Robust Data Augmentation for Few-Shot Learning

A Survey of Data Augmentation Approaches for NLP

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Empirical Analysis of Unlabeled Entity Problem in Named Entity Recognition

Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks

An Analysis of Simple Data Augmentation for Named Entity Recognition

The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT

GenAug: Data Augmentation for Finetuning Text Generators

Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space

Language Models are Few-Shot Learners

Enabling Language Models to Fill in the Blanks

POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training

MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

Paraphrase Augmented Task-Oriented Dialog Generation

Pretrained Transformers Improve Out-of-Distribution Robustness

Data Augmentation using Pre-trained Transformer Models

CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning

Blank Language Models

YAKE! Keyword extraction from single documents using multiple local features

Do Not Have Enough Data? Deep Learning to the Rescue!

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

XL-Editor: Post-editing Sentences with XLNet

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Plug and Play Language Models: A Simple Approach to Controlled Text Generation

Fine-Tuning Language Models from Human Preferences

CTRL: A Conditional Transformer Language Model for Controllable Generation

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Synthetic QA Corpora Generation with Roundtrip Consistency

TIGS: An Inference Algorithm for Text Infilling with Gradient Search

Augmenting Data with Mixup for Sentence Classification: An Empirical Study

MASS: Masked Sequence to Sequence Pre-training for Language Generation

EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks

Text Infilling

Conditional BERT Contextual Augmentation

Plan-And-Write: Towards Better Automatic Storytelling

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

A Note on the Inception Score

Decoupled Weight Decay Regularization

mixup: Beyond Empirical Risk Minimization

Data Augmentation for Morphological Reinflection

SQuAD: 100,000+ Questions for Machine Comprehension of Text

Improving Neural Machine Translation Models with Monolingual Data

Character-level Convolutional Networks for Text Classification

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Learning Word Vectors for Sentiment Analysis

Practical solutions to the problem of diagonal dominance in kernel document clustering

Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Bleu: a Method for Automatic Evaluation of Machine Translation

Catastrophic forgetting in connectionist networks

Long Short-Term Memory

A method for automatic evaluation of fault effects in the advanced intelligent network

Perplexity—a measure of the difficulty of speech recognition tasks

Sculpting Data for ML: The first act of Machine Learning

Parrot: Paraphrase generation for nlu

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Nlp chinese corpus: Large scale chi-nese corpus for nlp

From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition

contexts for existing entities. Table 12 shows the examples for SQuAD. Based on the target-aware sketch extraction, GENIUS can generate different new paragraphs according to the different questions

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL