Few-shot Reranking for Multi-hop QA via Language Model Prompting (2022-05-25T00:00:00.000000Z)

Abstract

We study few-shot reranking for multi-hop QA (MQA) with open-domain questions. To alleviate the need for a large number of labeled question-document pairs for retriever training, we propose PromptRank, which relies on language model prompting for multi-hop path reranking. PromptRank first constructs an instruction-based prompt that includes a candidate document path and then computes the relevance score between a given question and the path based on the conditional likelihood of the question given the path prompt according to a language model. PromptRank yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-the-art methods trained on thousands of examples — 73.6 recall@10 by PromptRank vs. 77.8 by PathRetriever and 77.5 by multi-hop dense retrieval.

Authors

Lu Wang

5 papers

Lajanugen Logeswaran

5 papers

Muhammad Khalifa

1 papers

Abstract

Authors

References54 items

OPT: Open Pre-trained Transformer Language Models

Improving Passage Retrieval with Zero-Shot Question Generation

Training language models to follow instructions with human feedback

Learning to Retrieve Passages without Supervision

Multitask Prompted Training Enables Zero-Shot Task Generalization

Finetuned Language Models Are Zero-Shot Learners

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Weakly Supervised Pre-Training for Multi-Hop Retriever

End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering

The Power of Scale for Parameter-Efficient Prompt Tuning

Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right

Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval

Calibrate Before Use: Improving Few-Shot Performance of Language Models

Making Pre-trained Language Models Better Few-shot Learners

HopRetriever: Retrieve Hops over Wikipedia to Answer Complex Questions

Few-Shot Text Generation with Natural Language Instructions

Distilling Knowledge from Reader to Retriever for Question Answering

How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering

Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps

Answering Open-Domain Questions of Varying Reasoning Steps from Text

RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs

Beyond [CLS] through Ranking by Generation

Transformers: State-of-the-Art Natural Language Processing

Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval

Language Models are Few-Shot Learners

Dense Passage Retrieval for Open-Domain Question Answering

Calibration of Pre-trained Transformers

Document Ranking with a Pretrained Sequence-to-Sequence Model

Differentiable Reasoning over a Virtual Knowledge Base

Unsupervised Question Decomposition for Question Answering

Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference

Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering

Chains-of-Reasoning at TextGraphs 2019 Shared Task: Reasoning over Chains of Facts for Explainable Multi-hop Inference

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration

Revealing the Importance of Semantic Retrieval for Machine Reading at Scale

Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answering

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Training a Ranking Function for Open-Domain Question Answering

On Calibration of Modern Neural Networks

Reading Wikipedia to Answer Open-Domain Questions

Billion-Scale Similarity Search with GPUs

Answering

Exploring the Limits

Relevance-Based Language Models

A Language Modeling Approach to Information Retrieval

Multitask Prompted Training Enables Zero-Shot Task Generalization

2021), where the top-100 retrieved paths are fed to the reader to obtain an answer from each path. Answers are then sorted based on a linear combination of path score and answer confidence

codebase.10 We use a maximum path length of 512 tokens, maximum question length of 64, and answer length of 30

ELECTRA: pre-training text encoders as discriminators rather than generators

Question Answering

GatfordCentre for Interactive Systems ResearchDepartment of Information

Search previous documents and ask the question. Question: 5 To analyze the documents and ask question

exhibits strong few-shot retrieval performance with as few as 128 examples and compares favorably to fully supervised methods (§3.1)

Field of Study

Venue Information

Name

Type

URL

Alternate Names