Learning To Retrieve Prompts for In-Context Learning

Published in

North American Chapter of the Association for C...(2021)

External Links:

Generate Graph DownloadPDF

TL;DR

This work proposes an efficient method for retrieving prompts for in-context learning using annotated data and an LM, and trains an efficient dense retriever from this data, which is used to retrieve training examples as prompts at test time.

Abstract

In-context learning is a recent paradigm in natural language understanding, where a large pre-trained language model (LM) observes a test instance and a few training examples as its input, and directly decodes the output without any update to its parameters. However, performance has been shown to strongly depend on the selected training examples (termed prompts). In this work, we propose an efficient method for retrieving prompts for in-context learning using annotated data and an LM. Given an input-output pair, we estimate the probability of the output given the input and a candidate training example as the prompt, and label training examples as positive or negative based on this probability. We then train an efficient dense retriever from this data, which is used to retrieve training examples as prompts at test time. We evaluate our approach on three sequence-to-sequence tasks where language utterances are mapped to meaning representations, and find that it substantially outperforms prior work and multiple baselines across the board.

Authors

Jonathan Berant

12 papers

Jonathan Herzig

4 papers

Ohad Rubin

2 papers

References63 items

Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Improving language models by retrieving from trillions of tokens

Learning To Retrieve Prompts for In-Context Learning

Published in

North American Chapter of the Association for C...(2021)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Jonathan Berant

12 papers

Jonathan Herzig

4 papers

Ohad Rubin

2 papers

References63 items

Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Data

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

Improving language models by retrieving from trillions of tokens

Scaling Language Models: Methods, Analysis & Insights from Training Gopher

Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention

An Explanation of In-context Learning as Implicit Bayesian Inference

Controllable Semantic Parsing via Retrieval Augmentation

The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design

Noisy Channel Language Model Prompting for Few-Shot Text Classification

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Evaluating Large Language Models Trained on Code

Case-based Reasoning for Natural Language Queries over Knowledge Bases

Constrained Language Models Yield Few-Shot Semantic Parsers

Question Decomposition with Dependency Graphs

Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right

Learning How to Ask: Querying LMs with Mixtures of Soft Prompts

Factual Probing Is [MASK]: Learning vs. Learning to Recall

GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow

BERTese: Learning to Speak to BERT

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

Calibrate Before Use: Improving Few-Shot Performance of Language Models

What Makes Good In-Context Examples for GPT-3?

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering

A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks

Nearest Neighbor Machine Translation

Task-Oriented Dialogue as Dataflow Synthesis

MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark

Language Models are Few-Shot Learners

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT

Dense Passage Retrieval for Open-Domain Question Answering

REALM: Retrieval-Augmented Language Model Pre-Training

Break It Down: A Question Understanding Benchmark

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference

Generalization through Memorization: Nearest Neighbor Language Models

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Language Models as Knowledge Bases?

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Latent Retrieval for Weakly Supervised Open Domain Question Answering

Efficient Natural Language Response Suggestion for Smart Reply

Reading Wikipedia to Answer Open-Domain Questions

Billion-Scale Similarity Search with GPUs

Adam: A Method for Stochastic Optimization

The Probabilistic Relevance Framework: BM25 and Beyond

OPTICS: ordering points to identify the clustering structure

Jurassic-1: Technical details and evaluation

GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Improving Evidence Retrieval for Automated Explainable Fact-Checking

McMillanMajor, and Shmargaret Shmitchell

Risk assessment Large language models have been shown to exhibit various kinds of bias

AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Improving the Cluster Structure Extracted from OPTICS Plots

Stochastic Neighbor Embedding

Yedid Hoshen, and Amnon Shashua. 2021. The inductive bias of in-context learning

obtained from such large LMs, it might also these biases. Additional examples

run training for 30 epochs. We used the default DPR hyperparameters without tuning

of the Association for Computational Linguistics

Retrieval-augmented generation for

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

North American Chapter of the Association for Computational Linguistics

Type

conference

URL

https://www.aclweb.org/portal/naacl

Alternate Names

North Am Chapter Assoc Comput Linguistics
NAACL