Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning (2018-08-11T00:00:00.000000Z)

TL;DR

Experiments show that D3Q significantly outperforms DDQ by controlling the quality of simulated experience used for planning and is further demonstrated in a domain extension setting, where the agent’s capability of adapting to a changing environment is tested.

Abstract

This paper presents a Discriminative Deep Dyna-Q (D3Q) approach to improving the effectiveness and robustness of Deep Dyna-Q (DDQ), a recently proposed framework that extends the Dyna-Q algorithm to integrate planning for task-completion dialogue policy learning. To obviate DDQ’s high dependency on the quality of simulated experiences, we incorporate an RNN-based discriminator in D3Q to differentiate simulated experience from real user experience in order to control the quality of training data. Experiments show that D3Q significantly outperforms DDQ by controlling the quality of simulated experience used for planning. The effectiveness and robustness of D3Q is further demonstrated in a domain extension setting, where the agent’s capability of adapting to a changing environment is tested.

Authors

Jianfeng Gao

39 papers

Yun-Nung (Vivian) Chen

8 papers

Jingjing Liu

13 papers

TL;DR

Abstract

Authors

References33 items

Tracking

Subgoal Discovery for Hierarchical Dialogue Policy Learning

Integrating planning for task-completion dialogue policy learning

Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning

Iterative policy learning in end-to-end trainable task-oriented neural dialog models

Imagination-Augmented Agents for Deep Reinforcement Learning

Sub-domain Modelling for Dialogue Management with Hierarchical Reinforcement Learning

Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning

End-to-End Task-Completion Neural Dialogue Systems

Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning

The Predictron: End-To-End Learning and Planning

A User Simulator for Task-Completion Dialogues

Multi-Domain Joint Semantic Frame Parsing Using Bi-Directional RNN-LSTM

Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access

Efficient Exploration for Dialog Policy Learning with Deep BBQ Networks \& Replay Buffer Spiking

BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-Oriented Dialogue Systems

Neural Belief Tracker: Data-Driven Dialogue State Tracking

Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning

Continuously Learning Neural Dialogue Management

Learning End-to-End Goal-Oriented Dialog

A Network-based End-to-End Trainable Task-oriented Dialogue System

Continuous Deep Q-Learning with Model-based Acceleration

Value Iteration Networks

Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval

Human-level control through deep reinforcement learning

POMDP-Based Statistical Spoken Dialog Systems: A Review

A survey on metrics for the evaluation of user simulations

Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System

Learning dialogue strategies within the Markov decision process framework

Efﬁcient Exploration for Dialogue Policy Learning with BBQ Networks & Replay Buffer Spiking

Neural networks for machine learning lecture 6a overview of mini-batch gradient descent

Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names