ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval (2023-05-18T00:00:00.000000Z)

TL;DR

A retrieval-enhanced framework to create training data from a general-domain unlabeled corpus for zero-shot learning, which achieves 4.3% gain over the strongest baselines and saves around 70% of the time compared to baselines using large NLG models.

Abstract

With the development of large language models (LLMs), zero-shot learning has attracted much attention for various NLP tasks. Different from prior works that generate training data with billion-scale natural language generation (NLG) models, we propose a retrieval-enhanced framework to create training data from a general-domain unlabeled corpus. To realize this, we first conduct contrastive pretraining to learn an unsupervised dense retriever for extracting the most relevant documents using class-descriptive verbalizers. We then further propose two simple strategies, namely Verbalizer Augmentation with Demonstrations and Self-consistency Guided Filtering to improve the topic coverage of the dataset while removing noisy examples. Experiments on nine datasets demonstrate that REGEN achieves 4.3% gain over the strongest baselines and saves around 70% of the time compared to baselines using large NLG models. Besides, REGEN can be naturally integrated with recently proposed large language models to boost performance.

Authors

Yu Meng

4 papers

Yuchen Zhuang

2 papers

Yue Yu

4 papers

TL;DR

Abstract

Authors

References104 items

Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation

Zero-Shot Listwise Document Reranking with a Large Language Model

The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Tasks

Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback

Neighborhood-Regularized Self-Training for Learning with Few Labels

Self-Instruct: Aligning Language Models with Self-Generated Instructions

Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer

Zero-Shot Text Classification with Self-Training

Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives

COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning

ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select

Don’t Prompt, Search! Mining-based Zero-Shot Learning with Language Models

ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback

Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach

Few-shot Learning with Retrieval Augmented Language Models

Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction

Training Subset Selection for Weak Supervision

Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning

Nearest Neighbor Zero-Shot Inference

Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

ZeroGen: Efficient Zero-shot Learning via Dataset Generation

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

How Can Graph Neural Networks Help Document Retrieval: A Case Study on CORD19 with Concept Map Generation

Learning To Retrieve Prompts for In-Context Learning

Unsupervised Dense Information Retrieval with Contrastive Learning

Improving language models by retrieving from trillions of tokens

Multitask Prompted Training Enables Zero-Shot Task Generalization

WRENCH: A Comprehensive Benchmark for Weak Supervision

NSP-BERT: A Prompt-based Few-Shot Learner through an Original Pre-training Task —— Next Sentence Prediction

Finetuned Language Models Are Zero-Shot Learners

Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback

Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval

Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integration

End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering

TaxoClass: Hierarchical Multi-Label Text Classification Using Only Class Names

True Few-Shot Learning with Language Models

SimCSE: Simple Contrastive Learning of Sentence Embeddings

Condenser: a Pre-training Architecture for Dense Retrieval

Generating Datasets with Pretrained Language Models

Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

Improving and Simplifying Pattern Exploiting Training

Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP

Calibrate Before Use: Improving Few-Shot Performance of Language Models

MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

Making Pre-trained Language Models Better Few-shot Learners

X-Class: Text Classification with Extremely Weak Supervision

Cold-start Active Learning through Self-Supervised Language Modeling

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

KILT: a Benchmark for Knowledge Intensive Language Tasks

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval

BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision

Language Models are Few-Shot Learners

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

SPECTER: Document-level Representation Learning using Citation-informed Transformers

Dense Passage Retrieval for Open-Domain Question Answering

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Generalization through Memorization: Nearest Neighbor Language Models

HuggingFace's Transformers: State-of-the-art Natural Language Processing

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Learning Dense Representations for Entity Retrieval

Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach

Discriminative Topic Mining via Category-Name Guided Text Embedding

RoBERTa: A Robustly Optimized BERT Pretraining Approach

When Does Label Smoothing Help?

Latent Retrieval for Weakly Supervised Open Domain Question Answering

Defending Against Neural Fake News

Texygen: A Benchmarking Platform for Text Generation Models

Billion-Scale Similarity Search with GPUs

Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering

Character-level Convolutional Networks for Text Classification

Hidden factors and hidden topics: understanding rating dimensions with review text

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank