A Large-Scale Chinese Short-Text Conversation Dataset
Published in
Natural Language Processing and Chinese Computing(2020)External Links:
TL
TL;DR
A large-scale cleaned Chinese conversation dataset, LCCC, which contains a base version (6.8million dialogues) and a large version (12.0 million dialogues), and pre-training dialogue models which are trained on LCCC-base and LCCC -large respectively.
Abstract
The advancements of neural dialogue generation models show promising results on modeling short-text conversations. However, training such models usually needs a large-scale high-quality dialogue corpus, which is hard to access. In this paper, we present a large-scale cleaned Chinese conversation dataset, LCCC, which contains a base version (6.8million dialogues) and a large version (12.0 million dialogues). The quality of our dataset is ensured by a rigorous data cleaning pipeline, which is built based on a set of rules and a classifier that is trained on manually annotated 110K dialogue pairs. We also release pre-training dialogue models which are trained on LCCC-base and LCCC-large respectively. The cleaned dataset and the pre-training models will facilitate the research of short-text conversation modeling. All the models and datasets are available at this https URL.
References43 items
1
Pre-trained models for natural language processing: A survey
2
An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation
3
Towards a Human-like Open-Domain Chatbot
4
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation
5