natural-language-processing-9

Open-Domain Dialog

3260 papers • 126 benchmarks • 313 datasets

This task has no description! Would you like to contribute one?

(Image credit: Papersgraph)

Benchmarks

These leaderboards are used to track progress in open-domain-dialog-9

Trend

Dataset

Best Model

Actions

KILT: Wizard of Wikipedia

Libraries

i

Use these libraries to find open-domain-dialog-9 models and implementations

Datasets

Subtasks

Dialogue Evaluation

Most implemented papers

KILT: a Benchmark for Knowledge Intensive Language Tasks

Yacine Jernite, F. Petroni, Aleksandra Piktus, Sebastian Riedel, Patrick Lewis, Angela Fan, Majid Yazdani, Nicola De Cao, James Thorne, Vassilis Plachouras, Tim Rocktaschel•Thu Sep 03 2020

It is found that a shared dense vector index coupled with a seq2seq model is a strong baseline, outperforming more tailor-made approaches for fact checking, open-domain question answering and dialogue, and yielding competitive results on entity linking and slot filling, by generating disambiguated text.

Content

DuLeMon

Reddit Conversation Corpus

ClovaCall

Diamante

701

0

Paper Graph

Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems

Àgata Lapedriza, Asma Ghandeharioun, Rosalind W. Picard, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah J. Jones•Fri May 31 2019

It is shown that this metric is capable of capturing the human-rated quality of a dialog model better than any automated metric known to-date, achieving a significant Pearson correlation (r>.7, p<.05).

94 0

Paper Graph

Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References

Jeffrey P. Bigham, Shikib Mehri, M. Eskénazi, Prakhar Gupta, Tiancheng Zhao, Amy Pavel•Sun Jun 30 2019

A series of experiments show that the use of multiple references results in improved correlation between several automatic metrics and human judgement for both the quality and the diversity of system output.

90 0

Paper Graph

Unsupervised Evaluation of Interactive Dialog with DialoGPT

Shikib Mehri, M. Eskénazi•Mon Jun 22 2020

The FED metric (fine-grained evaluation of dialog), an automatic evaluation metric which uses DialoGPT, without any fine-tuning or supervision is introduced, which attains moderate to strong correlation with human judgement at both levels.

201 0

Paper Graph

Dialogue Response Ranking Training with Large-Scale Human Feedback Data

Michel Galley, Chris Brockett, Bill Dolan, Yizhe Zhang, Xiang Gao•Mon Sep 14 2020

This work trained DialogRPT, a set of GPT-2 based models on 133M pairs of human feedback data and the resulting ranker outperformed several baselines, and outperforms the conventional dialog perplexity baseline with a large margin on predicting Reddit feedback.

115 0

Paper Graph

RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems

Dongyan Zhao, Rui Yan, Chongyang Tao, Lili Mou•Tue Jan 10 2017

RUBER is proposed, a Referenced metric and Unreferenced metrics Blended Evaluation Routine, which evaluates a reply by taking into consideration both a groundtruth reply and a query (previous user-issued utterance).

224 0

Paper Graph

Hurdles to Progress in Long-form Question Answering

Aurko Roy, Kalpesh Krishna, Mohit Iyyer•Tue Mar 09 2021

The task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress, and a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset is designed.

226 0

Paper Graph

Augmenting Neural Response Generation with Context-Aware Topical Attention

Nouha Dziri, Ehsan Kamalloo, K. Mathewson, Osmar R Zaiane•Thu Nov 01 2018

This work introduces a Topical Hierarchical Recurrent Encoder Decoder (THRED), a novel, fully data-driven, multi-turn response generation system intended to produce contextual and topic-aware responses.

84 0

Paper Graph

Evaluating Coherence in Dialogue Systems using Entailment

Nouha Dziri, Ehsan Kamalloo, K. Mathewson, Osmar R Zaiane•Fri Apr 05 2019

This paper presents interpretable metrics for evaluating topic coherence by making use of distributed sentence representations and introduces calculable approximations of human judgment based on conversational coherence by adopting state-of-the-art entailment techniques.

102 0

Paper Graph

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog

Àgata Lapedriza, S. Gu, Asma Ghandeharioun, Rosalind W. Picard, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah J. Jones•Sat Jun 29 2019

This work develops a novel class of off-policy batch RL algorithms, able to effectively learn offline, without exploring, from a fixed batch of human interaction data, using models pre-trained on data as a strong prior, and uses KL-control to penalize divergence from this prior during RL training.

377 0

Paper Graph

Adding a benchmark result helps the community track progress.