Deep Reinforcement Learning for Dialogue Generation

Published in

Conference on Empirical Methods in Natural Lang...(2016)

External Links:

Generate Graph DownloadPDF

TL;DR

This work simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity, non-repetitive turns, coherence, and ease of answering.

Abstract

Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes. Modeling the future direction of a dialogue is crucial to generating coherent, interesting dialogues, a need which led traditional NLP models of dialogue to draw on reinforcement learning. In this paper, we show how to integrate these goals, applying deep reinforcement learning to model future reward in chatbot dialogue. The model simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity (non-repetitive turns), coherence, and ease of answering (related to forward-looking function). We evaluate our model on diversity, length as well as with human judges, showing that the proposed algorithm generates more interactive responses and manages to foster a more sustained conversation in dialogue simulation. This work marks a first step towards learning a neural conversational model based on the long-term success of dialogues.

Authors

Dan Jurafsky

15 papers

Jiwei Li

2 papers

Will Monroe

3 papers

References57 items

Continuously Learning Neural Dialogue Management

Incorporating loose-structured knowledge into conversation modeling via recall-gate LSTM

Incorporating Loose-Structured Knowledge into LSTM with Recall Gate for Conversation Modeling

A Network-based End-to-End Trainable Task-oriented Dialogue System

LSTM based Conversation Models

Deep Reinforcement Learning for Dialogue Generation

Published in

Conference on Empirical Methods in Natural Lang...(2016)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Dan Jurafsky

15 papers

Jiwei Li

2 papers

Will Monroe

3 papers

References57 items

Continuously Learning Neural Dialogue Management

Incorporating loose-structured knowledge into conversation modeling via recall-gate LSTM

Incorporating Loose-Structured Knowledge into LSTM with Recall Gate for Conversation Modeling

A Network-based End-to-End Trainable Task-oriented Dialogue System

LSTM based Conversation Models

Alan Ritter

11 papers

Michel Galley

8 papers

Jianfeng Gao

39 papers

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

A Persona-Based Neural Conversation Model

Mastering the game of Go with deep neural networks and tree search

A Survey of Available Corpora for Building Data-Driven Dialogue Systems

Sequence Level Training with Recurrent Neural Networks

Deep Reinforcement Learning with a Natural Language Action Space

Deep Reinforcement Learning with an Action Space Defined by Natural Language

Attention with Intention for a Neural Network Conversation Model

A Diversity-Promoting Objective Function for Neural Conversation Models

Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems

Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models

Hierarchical Neural Network Generative Models for Movie Dialogues

Language Understanding for Text-based Games using Deep Reinforcement Learning

deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

A Neural Network Approach to Context-Sensitive Generation of Conversational Responses

A Neural Conversational Model

Reinforcement Learning Neural Turing Machines

Reinforcement Learning Neural Turing Machines - Revised

Neural Responding Machine for Short-Text Conversation

Sequence to Sequence Learning with Neural Networks

Neural Machine Translation by Jointly Learning to Align and Translate

Playing Atari with Deep Reinforcement Learning

On-line policy optimisation of Bayesian spoken dialogue systems via human interaction

POMDP-Based Statistical Spoken Dialog Systems: A Review

IRIS: a Chat-oriented Dialogue System based on the Vector Space Model

Data-Driven Response Generation in Social Media

Learning to Win by Reading Manuals in a Monte-Carlo Framework

Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System

Learning to Follow Navigational Directions

The Hidden Information State model: A practical framework for POMDP-based spoken dialogue management

Are We There Yet? Research in Commercial Spoken Dialog Systems

Curriculum learning

A trainable generator for recommendations in multimodal dialog

Bleu: a Method for Automatic Evaluation of Machine Translation

Trainable approaches to surface natural language generation and their application to conversational dialog systems

Empirical Evaluation of a Reinforcement Learning Spoken Dialogue System

Stochastic Language Generation for Spoken Dialogue Systems

An Application of Reinforcement Learning to Dialogue Strategy Selection in a Spoken Dialogue System for Email

Reinforcement Learning for Spoken Dialogue Systems

Policy Gradient Methods for Reinforcement Learning with Function Approximation

Learning dialogue strategies within the Markov decision process framework

Likelihood ratio gradient estimation for stochastic systems

Incremental on-line adaptation of POMDP-based dialogue managers to extended domains

POMDP-based dialogue manager adaptation to extended domains

Developing Non-goal Dialog System Based on Examples of Drama Television

Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Stochastic Optimization

A stochastic model of human-machine interaction for learning dialog strategies

On the Semantics and Pragmatics of Linguistic Feedback

Opening up Closings

DOI: 10.1017/S000000000000000 Printed in the United Kingdom A Survey of Statistical User Simulation Techniques for Reinforcement-Learning of Dialogue Management Strategies

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/1606.01541

Venue Information

Name

Conference on Empirical Methods in Natural Language Processing

Type

conference

URL

https://www.aclweb.org/portal/emnlp

Alternate Names

Empir Method Nat Lang Process
Empirical Methods in Natural Language Processing
Conf Empir Method Nat Lang Process
EMNLP