natural-language-processing

Dialogue Evaluation

3260 papers • 126 benchmarks • 313 datasets

This task has no description! Would you like to contribute one?

(Image credit: Papersgraph)

Benchmarks

These leaderboards are used to track progress in dialogue-evaluation

Trend

Dataset

Best Model

Actions

USR-TopicalChat

USR-PersonaChat

Libraries

i

Use these libraries to find dialogue-evaluation models and implementations

Datasets

FaithDial

USR-TopicalChat

USR-PersonaChat

Reddit Engagement Dataset

SaGA

Subtasks

No subtasks available.

Most implemented papers

Adversarial Learning for Neural Dialogue Generation

Dan Jurafsky, Jiwei Li, Will Monroe, Alan Ritter, Tianlin Shi, Sébastien Jean•Sun Jan 22 2017

This work applies adversarial training to open-domain dialogue generation, training a system to produce sequences that are indistinguishable from human-generated dialogue utterances, and investigates models for adversarial evaluation that uses success in fooling an adversary as a dialogue evaluation metric, while avoiding a number of potential pitfalls.

912

Content

0

Paper Graph

Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References

Jeffrey P. Bigham, Shikib Mehri, M. Eskénazi, Prakhar Gupta, Tiancheng Zhao, Amy Pavel•Sun Jun 30 2019

A series of experiments show that the use of multiple references results in improved correlation between several automatic metrics and human judgement for both the quality and the diversity of system output.

90 0

Paper Graph

Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems

Àgata Lapedriza, Asma Ghandeharioun, Rosalind W. Picard, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah J. Jones•Fri May 31 2019

It is shown that this metric is capable of capturing the human-rated quality of a dialog model better than any automated metric known to-date, achieving a significant Pearson correlation (r>.7, p<.05).

94 0

Paper Graph

Unsupervised Evaluation of Interactive Dialog with DialoGPT

Shikib Mehri, M. Eskénazi•Mon Jun 22 2020

The FED metric (fine-grained evaluation of dialog), an automatic evaluation metric which uses DialoGPT, without any fine-tuning or supervision is introduced, which attains moderate to strong correlation with human judgement at both levels.

201 0

Paper Graph

Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems

A. Galstyan, Nanyun Peng, Sarik Ghazarian, R. Weischedel•Sun Nov 03 2019

It is demonstrated that human annotators have high agreement on assessing utterance-level engagement scores and that these scores can improve automatic evaluation metrics for open-domain dialogue systems, as shown by correlation with human judgements.

58 0

Paper Graph

Automatic Evaluation and Moderation of Open-domain Dialogue Systems

Alexander I. Rudnicky, João Sedoc, Chen Zhang, L. F. D’Haro, Rafael E. Banchs•Tue Nov 02 2021

This paper describes the datasets and baselines provided to participants, as well as submission evaluation results for each of the two proposed subtasks, and describes the automatic evaluation mechanisms that show high correlations with human judgements across multiple dialogue evaluation aspects.

41 0

Paper Graph

RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems

Dongyan Zhao, Rui Yan, Chongyang Tao, Lili Mou•Tue Jan 10 2017

RUBER is proposed, a Referenced metric and Unreferenced metrics Blended Evaluation Routine, which evaluates a reply by taking into consideration both a groundtruth reply and a query (previous user-issued utterance).

224 0

Paper Graph

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Joelle Pineau, Yoshua Bengio, Ryan Lowe, Iulian Serban, Michael Noseworthy, Nicolas Angelard-Gontier•Thu Feb 16 2017

An evaluation model that learns to predict human-like scores to input responses, using a new dataset of human response scores is presented and it is shown that the ADEM model’s predictions correlate significantly, and at a level much higher than word-overlap metrics such as BLEU, with human judgements at both the utterance and system-level.

381 0

Paper Graph

FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation

Chen Zhang, L. F. D’Haro, Qiquan Zhang, Thomas Friedrichs, Haizhou Li•Mon Oct 24 2022

A multi-dimensional dialogue-level metric, which consists of three sub-metrics with each targeting a specific dimension, which is trained with novel self-supervised objectives and exhibit strong correlations with human judgment for their respective dimensions.

22 0

Paper Graph

Don’t Forget Your ABC’s: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems

Jinho D. Choi, Sarah E. Finch, James D. Finch•Sat Dec 17 2022

A novel human evaluation method to estimate the rates of many{pasted macro ‘LN’} dialogue system behaviors is presented and it is demonstrated that this behavior method is more suitable than alternative Likert-style or comparative approaches for dimensional evaluation of these systems.

15 0

Paper Graph

Adding a benchmark result helps the community track progress.