How to Build User Simulators to Train RL-based Dialog Systems (2019-09-03T00:00:00.000000Z)

TL;DR

A method of standardizing user simulator building is proposed that can be used by the community to compare dialog system quality using the same set of user simulators fairly and asks human users to assess the simulators directly and indirectly by rating the simulated dialogs and interacting with the trained systems.

Abstract

User simulators are essential for training reinforcement learning (RL) based dialog models. The performance of the simulator directly impacts the RL policy. However, building a good user simulator that models real user behaviors is challenging. We propose a method of standardizing user simulator building that can be used by the community to compare dialog system quality using the same set of user simulators fairly. We present implementations of six user simulators trained with different dialog planning and generation methods. We then calculate a set of automatic metrics to evaluate the quality of these simulators both directly and indirectly. We also ask human users to assess the simulators directly and indirectly by rating the simulated dialogs and interacting with the trained systems. This paper presents a comprehensive evaluation framework for user simulator study and provides a better understanding of the pros and cons of different user simulators, as well as their impacts on the trained systems.

Authors

Weiyan Shi

3 papers

Zhou Yu

4 papers

Xuewei Wang

2 papers

TL;DR

Abstract

Authors

References46 items

Domain Adaptive Dialog Generation via Meta Learning

ConvLab: Multi-Domain End-to-End Dialog System Platform

Unsupervised Dialog Structure Learning

Unifying Human and Statistical Evaluation for Natural Language Generation

Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models

MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling

Decoupling Strategy and Generation in Negotiation Dialogues

Adversarial Domain Adaptation for Variational Neural Language Generation in Dialogue Systems

Sequicity: Simplifying Task-oriented Dialogue Systems with Single Sequence-to-Sequence Architectures

Neural User Simulation for Corpus-based Policy Optimisation of Spoken Dialogue Systems

Sentiment Adaptive End-to-End Dialog Systems

Building a Conversational Agent Overnight with Dialogue Self-Play

Iterative policy learning in end-to-end trainable task-oriented neural dialog models

Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management

Semantic Refinement GRU-Based Neural Language Generation for Spoken Dialogue Systems

ParlAI: A Dialog Research Software Platform

Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning

A User Simulator for Task-Completion Dialogues

Sequential Match Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots

Dialogue manager domain adaptation using Gaussian process reinforcement learning

A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems

Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

Strategic Dialogue Management via Deep Reinforcement Learning

Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems

Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking

Convolutional Neural Network Architectures for Matching Natural Language Sentences

The third Dialog State Tracking Challenge

Sequence to Sequence Learning with Neural Networks

POMDP-Based Statistical Spoken Dialog Systems: A Review

A survey on metrics for the evaluation of user simulations

Comparing user simulations for dialogue strategy learning

Assessing user simulation for dialog systems using human judges and automatic evaluation measures

Adaptive epsilon-Greedy Exploration in Reinforcement Learning Based on Value Difference

Analysis of a new simulation approach to dialog system evaluation

Data-driven user simulation for automated evaluation of spoken dialog systems

The Hidden Agenda User Simulation Model

Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System

Effects of the user model on simulation-based learning of dialogue strategies

Let's go public! taking a spoken dialog system to the real world

Bleu: a Method for Automatic Evaluation of Machine Translation

User Modeling and User-Adapted Interaction

Adaptive ε-greedy Exploration in Reinforcement Learning Based on Value Differences

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Spoken dialogue systems

DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies

Field of Study

Venue Information

Name

Type

URL

Alternate Names