Prompting Large Language Model for Machine Translation: A Case Study (2023-01-17T00:00:00.000000Z)

TL;DR

A systematic study on prompting strategies for translation, examining various factors for prompt template and demonstration example selection, and exploring the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning in prompting.

Abstract

Research on prompting has shown excellent performance with little or even no supervised training across many tasks. However, prompting for machine translation is still under-explored in the literature. We fill this gap by offering a systematic study on prompting strategies for translation, examining various factors for prompt template and demonstration example selection. We further explore the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning in prompting. Extensive experiments with GLM-130B (Zeng et al., 2022) as the testbed show that 1) the number and the quality of prompt examples matter, where using suboptimal examples degenerates translation; 2) several features of prompt examples, such as semantic similarity, show significant Spearman correlation with their prompting performance; yet, none of the correlations are strong enough; 3) using pseudo parallel prompt examples constructed from monolingual data via zero-shot prompting could improve translation; and 4) improved performance is achievable by transferring knowledge from prompt examples selected in other settings. We finally provide an analysis on the model outputs and discuss several problems that prompting still suffers from.

Authors

Biao Zhang

1 papers

B. Haddow

9 papers

Alexandra Birch

1 papers

TL;DR

Abstract

Authors

References66 items

Extrinsic Evaluation of Machine Translation Metrics

In-context Examples Selection for Machine Translation

Prompting PaLM for Translation: Assessing Strategies and Performance

Active Example Selection for In-Context Learning

Scaling Instruction-Finetuned Language Models

GLM-130B: An Open Bilingual Pre-trained Model

Complexity-Based Prompting for Multi-Step Reasoning

Bidirectional Language Models Are Also Few-shot Learners

News Summarization and Evaluation in the Era of GPT-3

Out of One, Many: Using Language Models to Simulate Human Samples

No Language Left Behind: Scaling Human-Centered Machine Translation

Emergent Abilities of Large Language Models

Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages

Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

OPT: Open Pre-trained Transformer Language Models

PaLM: Scaling Language Modeling with Pathways

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

Using natural language prompts for machine translation

Chain of Thought Prompting Elicits Reasoning in Large Language Models

MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators

Unsupervised Neural Machine Translation with Generative Language Models Only

The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21

Finetuned Language Models Are Zero-Shot Learners

WeChat Neural Machine Translation Systems for WMT21

Calibrate Before Use: Improving Few-Shot Performance of Language Models

Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm

What Makes Good In-Context Examples for GPT-3?

Making Pre-trained Language Models Better Few-shot Learners

mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

Rethinking Document-level Neural Machine Translation

COMET: A Neural Framework for MT Evaluation

Language Models are Few-Shot Learners

Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

Unsupervised Domain Clusters in Pretrained Language Models

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

Scaling Laws for Neural Language Models

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges

WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

Generating Diverse Translations with Sentence Codes

Tagged Back-Translation

Search Engine Guided Neural Machine Translation

A Call for Clarity in Reporting BLEU Scores

Guiding Neural Machine Translation with Retrieved Translation Pieces

Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Exploiting Source-side Monolingual Data in Neural Machine Translation

Improving Neural Machine Translation Models with Monolingual Data

Prompt-Driven Neural Machine Translation

2022c) developed chain-of-thought (CoT

SemScore semantic score based on the cosine similarity of the demonstration’s source and target sentence embed-dings from LASER2

Findings of the 2021 Conference on Machine Translation (WMT21)

TransPrompt: Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification

Tencent Translation System for the WMT21 News Translation Task

Few-shot Learning with Multilingual Language Models

for Machine Translation: A Case Study 2590, Computational findings-acl.

Large Language Model

MTScore translation quality of the prompt example from COMET QE wmt20-comet-qe-da

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

2019. BERT: Pre-training

Controlling Politeness in Neural Machine Translation via Side Constraints

MT evaluation

CaseSemScore-Tgt shows the highest correlation on average followed by TLength and SemScore

LMScore GLM-130B-based, length-normalized log likelihood of the demonstration

50 1-shot Translation (Low-quality Pool

of the Association for Computational Linguistics

Edinburgh Research Explorer Prompting Large Language Model for Machine Translation: A Case Study

CaseSemScore-Src similarity to the input that averages over SemScores between the test input and the demon-stration’s source

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names