goal-oriented-dialog

User Simulation

3260 papers • 126 benchmarks • 313 datasets

Simulation of user interacting with a dialog system

(Image credit: Papersgraph)

Benchmarks

These leaderboards are used to track progress in goal-oriented-dialog

Trend

Dataset

Best Model

Actions

No benchmarks available.

Libraries

i

Use these libraries to find goal-oriented-dialog models and implementations

Datasets

No datasets available.

Subtasks

No subtasks available.

Most implemented papers

KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems

Xiangnan He, Tat-seng Chua, Wenqiang Lei, Chongming Gao, Shijun Li, Jiawei Chen, Biao Li, Peng Jiang, Jiaxin Mao•Mon Feb 21 2022

A fully-observed dataset from Kuaishou's online environment, where almost all 1,411 users have been exposed to all 3,327 items, and a preliminary analysis of how the two factors - data density and exposure bias - affect the evaluation results of multi-round conversational recommendation.

192

Content

0

Paper Graph

A User Simulator for Task-Completion Dialogues

Zachary Chase Lipton, Jianfeng Gao, Yun-Nung (Vivian) Chen, Bhuwan Dhingra, Xiujun Li, Lihong Li•Fri Dec 16 2016

A new, publicly available simulation framework, where the simulator, designed for the movie-booking domain, leverages both rules and collected data, and several agents are demonstrated and the procedure to add and test your own agent is detailed.

176 0

Paper Graph

CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset

Minlie Huang, Kaili Huang, Xiaoyan Zhu, Qi Zhu, Zheng Zhang•Wed Feb 26 2020

The large size and rich annotation of CrossWOZ make it suitable to investigate a variety of tasks in cross-domain dialogue modeling, such as dialogue state tracking, policy learning, user simulation, etc.

117 0

Paper Graph

In-Context Learning User Simulators for Task-Oriented Dialog Systems

Silvia Terragni, Modestas Filipavicius, Nghia Khau, Bruna Guedes, A. Manso, Roland Mathis•Wed May 31 2023

A novel application of large language models in user simulation for task-oriented dialog systems, specifically focusing on an in-context learning approach, which eliminates the need for labor-intensive rule definition or extensive annotated data, making it more efficient and accessible.

18 0

Paper Graph

Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems

K. Balog, M. de Rijke, Shuo Zhang, Weiwei Sun, Pengjie Ren, Zhumin Chen, Z. Ren•Fri May 07 2021

A user satisfaction annotation dataset, USS, is proposed that includes 6,800 dialogues sampled from multiple domains, spanning real-world e-commerce dialogues, task-oriented dialogues constructed through Wizard-of-Oz experiments, and movie recommendation dialogues and three baseline methods for user satisfaction prediction and action prediction tasks.

70 0

Paper Graph

Evaluating Conversational Recommender Systems via User Simulation

K. Balog, Shuo Zhang•Sun Jun 14 2020

The user simulator aims to generate responses that a real human would give by considering both individual preferences and the general flow of interaction with the system, and shows that preference modeling and task-specific interaction models both contribute to more realistic simulations.

124 0

Paper Graph

Reactive and Safe Road User Simulations using Neural Barrier Certificates

Yue Meng, Zengyi Qin, Chuchu Fan•Mon Sep 13 2021

This work proposed a reactive agent model which can ensure safety without comprising the original purposes, by learning only high-level decisions from expert data and a low level decentralized controller guided by the jointly learned decentralized barrier certificates.

20 0

Paper Graph

Evaluating Mixed-initiative Conversational Search Systems via User Simulation

F. Crestani, Mohammad Aliannejadi, Ivan Sekulic•Thu Feb 10 2022

This paper proposes a conversational User Simulator, called USi, for automatic evaluation of conversational search systems, capable of automatically answering clarifying questions about the topic throughout the search session, and shows that responses generated by USi are both inline with the underlying information need and comparable to human-generated answers.

64 0

Paper Graph

Variational Hierarchical Dialog Autoencoder for Dialog State Tracking Data Augmentation

Kang Min Yoo, Franck Dernoncourt, Trung Bui, W. Chang, Sang-goo Lee, Hanbit Lee•Wed Jan 22 2020

This work proposes the Variational Hierarchical Dialog Autoencoder (VHDA) for modeling complete aspects of goal-oriented dialogs, including linguistic features and underlying structured annotations, namely dialog acts and goals, and proposes two training policies to mitigate issues that arise with training VAE-based models.

25 0

Paper Graph

How to Build User Simulators to Train RL-based Dialog Systems

Weiyan Shi, Zhou Yu, Xuewei Wang, Kun Qian•Mon Sep 02 2019

A method of standardizing user simulator building is proposed that can be used by the community to compare dialog system quality using the same set of user simulators fairly and asks human users to assess the simulators directly and indirectly by rating the simulated dialogs and interacting with the trained systems.

70 0

Paper Graph

Adding a benchmark result helps the community track progress.