MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
TL
TL;DR
This paper investigates strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality and validate the new metric, namely MoverScore, on a number of text generation tasks.
Abstract
A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.
References55 items
1
Better Rewards Yield Better Summaries: Learning to Summarise Without References
2
Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation
3
Studying Summarization Evaluation Metrics in the Appropriate Scoring Range
4
Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts
5