A Large-Scale Corpus for Conversation Disentanglement
TL
TL;DR
A new dataset of 77,563 messages manually annotated with reply-structure graphs that both disentangle conversations and define internal conversation structure is created, which is 16 times larger than all previously released datasets combined, the first to include adjudication of annotation disagreements, and theFirst to include context.
Abstract
Disentangling conversations mixed together in a single stream of messages is a difficult task, made harder by the lack of large manually annotated datasets. We created a new dataset of 77,563 messages manually annotated with reply-structure graphs that both disentangle conversations and define internal conversation structure. Our data is 16 times larger than all previously released datasets combined, the first to include adjudication of annotation disagreements, and the first to include context. We use our data to re-examine prior work, in particular, finding that 89% of conversations in a widely used dialogue corpus are either missing messages or contain extra messages. Our manually-annotated data presents an opportunity to develop robust data-driven methods for conversation disentanglement, which will help advance dialogue research.
References41 items
1
Recovering Implicit Thread Structure in Newsgroup Style Conversations
2
SLATE: A Super-Lightweight Annotation Tool for Experts
3
Dialog System Technology Challenge 7
4
Learning to Disentangle Interleaved Conversational Threads with a Siamese Hierarchical Network and Similarity Ranking
5