Evaluating Natural Language Understanding Services for Conversational Question Answering Systems
TL
TL;DR
This paper presents a method to evaluate the classification performance of NLU services, and presents two new corpora, one consisting of annotated questions with the corresponding answers, to enable both, researchers and companies to make more educated decisions about which service they should use.
Abstract
Conversational interfaces recently gained a lot of attention. One of the reasons for the current hype is the fact that chatbots (one particularly popular form of conversational interfaces) nowadays can be created without any programming knowledge, thanks to different toolkits and so-called Natural Language Understanding (NLU) services. While these NLU services are already widely used in both, industry and science, so far, they have not been analysed systematically. In this paper, we present a method to evaluate the classification performance of NLU services. Moreover, we present two new corpora, one consisting of annotated questions and one consisting of annotated questions with the corresponding answers. Based on these corpora, we conduct an evaluation of some of the most popular NLU services. Thereby we want to enable both, researchers and companies to make more educated decisions about which service they should use.
References19 items
1
Applying Chatbots to the Internet of Things: Opportunities and Architectural Elements
2
Rapid Prototyping of Form-driven Dialogue Systems Using an Open-source Framework
3
The Conversational Interface: Talking to Smart Devices
4
NLU vs. Dialog Management: To Whom am I Speaking?
5