BERTScore: Evaluating Text Generation with BERT (2019-04-21T00:00:00.000000Z)

TL;DR

This work proposes BERTScore, an automatic evaluation metric for text generation that correlates better with human judgments and provides stronger model selection performance than existing metrics.

Abstract

We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.

Authors

Tianyi Zhang

4 papers

Felix Wu

4 papers

Kilian Q. Weinberger

7 papers

TL;DR

Abstract

Authors

References105 items

Regression Analysis

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources

WMDO: Fluency-based Word Mover’s Distance for Machine Translation Evaluation

Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization

EED: Extended Edit Distance Measure for Machine Translation

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow

Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation

Beyond BLEU:Training Neural Machine Translation with Semantic Similarity

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts

Alternative Weighting Schemes for ELMo Embeddings

Unifying Human and Statistical Evaluation for Natural Language Generation

PAWS: Paraphrase Adversaries from Word Scrambling

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

ANA at SemEval-2019 Task 3: Contextual Emotion detection in Conversations through hierarchical LSTMs and BERT

Simple Applications of BERT for Ad Hoc Document Retrieval

Fine-tune BERT for Extractive Summarization

A context-aware citation recommendation model with BERT and graph convolutional networks

Linguistic Knowledge and Transferability of Contextual Representations

SciBERT: A Pretrained Language Model for Scientific Text

Pay Less Attention with Lightweight and Dynamic Convolutions

Cross-lingual Language Model Pretraining

RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation

Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance

The price of debiasing automatic metrics in natural language evalaution

Probabilistic FastText for Multi-Sense Word Embeddings

Learning to Evaluate Image Captioning

Scaling Neural Machine Translation

A Call for Clarity in Reporting BLEU Scores

Tensor2Tensor for Neural Machine Translation

Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations

Learning Word Vectors for 157 Languages

Deep Contextualized Word Representations

Natural Language Inference over Interaction Space

MEANT 2.0: Accurate semantic MT evaluation for any output language

Results of the WMT17 Metrics Shared Task

Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation

Attention is All you Need

A Mixture Model for Learning Multi-Sense Word Embeddings

Convolutional Sequence to Sequence Learning

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Results of the WMT16 Metrics Shared Task

SPICE: Semantic Propositional Image Caption Evaluation

A Decomposable Attention Model for Natural Language Inference

Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics

chrF: character n-gram F-score for automatic MT evaluation

From Word Embeddings To Document Distances

deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

CIDEr: Consensus-based image description evaluation

GloVe: Global Vectors for Word Representation

Testing for Significance of Increased Correlation with Human Judgment

BEER: BEtter Evaluation as Ranking

Meteor Universal: Language Specific Translation Evaluation for Any Target Language

Microsoft COCO: Common Objects in Context

Distributed Representations of Words and Phrases and their Compositionality

A Comparison of Greedy and Optimal Assessment of Natural Language Student Input Using Word-to-Word Similarity Metrics

Automatic Evaluation of Translation Quality for Distant Language Pairs

Fluency, Adequacy, or HTER? Exploring Different Human Judgments with a Tunable MT Metric

Moses: Open Source Toolkit for Statistical Machine Translation

CDER: Efficient MT Evaluation Using Block Movements

Measuring the Semantic Similarity of Texts

METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

ROUGE: A Package for Automatic Evaluation of Summaries

Testing the correlation of word error rate and perplexity

Bleu: a Method for Automatic Evaluation of Machine Translation

Automatic evaluation of machine translation quality using n-gram co-occurrence statistics

A metric for distributions with applications to image databases

Accelerated DP based search for statistical translation

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Filtering Pseudo-References by Paraphrasing for Automatic Evaluation of Machine Translation

Meteor++ 2.0: Adopt Syntactic Level Paraphrase Knowledge into Machine Translation Evaluation

2019) uses the embeddings from the last 5 layers from BERT and L2-normalizes