USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
TL
TL;DR
USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog and is shown to strongly correlate with human judgment on both Topical-Chat and PersonaChat.
Abstract
The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for evaluating dialog models. To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog. USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog. USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, system-level: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally produces interpretable measures for several desirable properties of dialog.
References30 items
1
HuggingFace's Transformers: State-of-the-art Natural Language Processing
2
Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations
3
RoBERTa: A Robustly Optimized BERT Pretraining Approach
4
Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References
5