MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

Published in

Conference on Empirical Methods in Natural Lang...(2019)

External Links:

Generate Graph DownloadPDF

TL;DR

This paper investigates strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality and validate the new metric, namely MoverScore, on a number of text generation tasks.

Abstract

A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.

Authors

Christian M. Meyer

3 papers

Fei Liu

4 papers

Maxime Peyrard

4 papers

References55 items

Better Rewards Yield Better Summaries: Learning to Summarise Without References

Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation

Studying Summarization Evaluation Metrics in the Appropriate Scoring Range

Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts

Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications

MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

Published in

Conference on Empirical Methods in Natural Lang...(2019)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Christian M. Meyer

3 papers

Fei Liu

4 papers

Maxime Peyrard

4 papers

References55 items

Better Rewards Yield Better Summaries: Learning to Summarise Without References

Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation

Studying Summarization Evaluation Metrics in the Appropriate Scoring Range

Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts

Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications

Steffen Eger

3 papers

What do you learn from context? Probing for sentence structure in contextualized word representations

BERTScore: Evaluating Text Generation with BERT

Linguistic Knowledge and Transferability of Contextual Representations

Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems

RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation

A Structured Review of the Validity of BLEU

Learning Neural Templates for Text Generation

The price of debiasing automatic metrics in natural language evalaution

Learning to Evaluate Image Captioning

Fast dynamic routing based on weighted kernel density estimation

Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies

A Call for Clarity in Reporting BLEU Scores

Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Investigating Capsule Networks with Dynamic Routing for Text Classification

Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations

Deep Contextualized Word Representations

A Simple Theoretical Model of Importance for Summarization

Results of the WMT17 Metrics Shared Task

MEANT 2.0: Accurate semantic MT evaluation for any output language

Learning to Score System Summaries for Better Content Selection Evaluation.

Why We Need New Evaluation Metrics for NLG

Get To The Point: Summarization with Pointer-Generator Networks

Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

SPICE: Semantic Propositional Image Caption Evaluation

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

A Diversity-Promoting Objective Function for Neural Conversation Models

Better Summarization Evaluation with Word Embeddings for ROUGE

Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems

From Word Embeddings To Document Distances

Teaching Machines to Read and Comprehend

CIDEr: Consensus-based image description evaluation

Results of the WMT14 Metrics Shared Task

Microsoft COCO: Common Objects in Context

Automatically Assessing Machine Summary Content Without a Gold Standard

METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

Automated Summarization Evaluation with Basic Elements.

ROUGE: A Package for Automatic Evaluation of Summaries

Bleu: a Method for Automatic Evaluation of Machine Translation

Mean Shift: A Robust Approach Toward Feature Space Analysis

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Meteor++: Incorporating Copy Knowledge into Machine Translation Evaluation

The 54th Annual Meeting of the Association for Computational Linguistics

Phrase-based statistical language generation

Summarization Evaluation Using Transformed Basic Elements

Evaluating Content Selection in Summarization: The Pyramid Method

c ○ 2000 Kluwer Academic Publishers. Manufactured in The Netherlands. The Earth Mover’s Distance as a Metric for Image Retrieval

using graphical models and active learning

Field of Study

Computer Science

Venue Information

Name

Conference on Empirical Methods in Natural Language Processing

Type

conference

URL

https://www.aclweb.org/portal/emnlp

Alternate Names

Empir Method Nat Lang Process
Empirical Methods in Natural Language Processing
Conf Empir Method Nat Lang Process
EMNLP

TL;DR

Abstract

Authors

References55 items

Better Rewards Yield Better Summaries: Learning to Summarise Without References

Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation

Studying Summarization Evaluation Metrics in the Appropriate Scoring Range

Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts

Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications

TL;DR

Abstract

Authors

References55 items

Better Rewards Yield Better Summaries: Learning to Summarise Without References

Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation

Studying Summarization Evaluation Metrics in the Appropriate Scoring Range

Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts

Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications

What do you learn from context? Probing for sentence structure in contextualized word representations

BERTScore: Evaluating Text Generation with BERT

Linguistic Knowledge and Transferability of Contextual Representations

Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems

RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation

A Structured Review of the Validity of BLEU

Learning Neural Templates for Text Generation

The price of debiasing automatic metrics in natural language evalaution

Learning to Evaluate Image Captioning

Fast dynamic routing based on weighted kernel density estimation

Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies

A Call for Clarity in Reporting BLEU Scores

Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Investigating Capsule Networks with Dynamic Routing for Text Classification

Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations

Deep Contextualized Word Representations

A Simple Theoretical Model of Importance for Summarization

Results of the WMT17 Metrics Shared Task

MEANT 2.0: Accurate semantic MT evaluation for any output language

Learning to Score System Summaries for Better Content Selection Evaluation.

Why We Need New Evaluation Metrics for NLG

Get To The Point: Summarization with Pointer-Generator Networks

Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

SPICE: Semantic Propositional Image Caption Evaluation

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

A Diversity-Promoting Objective Function for Neural Conversation Models

Better Summarization Evaluation with Word Embeddings for ROUGE

Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems

From Word Embeddings To Document Distances

Teaching Machines to Read and Comprehend

CIDEr: Consensus-based image description evaluation

Results of the WMT14 Metrics Shared Task

Microsoft COCO: Common Objects in Context

Automatically Assessing Machine Summary Content Without a Gold Standard

METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

Automated Summarization Evaluation with Basic Elements.

ROUGE: A Package for Automatic Evaluation of Summaries

Bleu: a Method for Automatic Evaluation of Machine Translation

Mean Shift: A Robust Approach Toward Feature Space Analysis

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Meteor++: Incorporating Copy Knowledge into Machine Translation Evaluation

The 54th Annual Meeting of the Association for Computational Linguistics

Phrase-based statistical language generation

Summarization Evaluation Using Transformed Basic Elements

Evaluating Content Selection in Summarization: The Pyramid Method

Kernel Smoothing

c ○ 2000 Kluwer Academic Publishers. Manufactured in The Netherlands. The Earth Mover’s Distance as a Metric for Image Retrieval

Suofei

using graphical models and active learning

Field of Study

Venue Information

Name

Type

URL

Alternate Names