The WiLI benchmark dataset for written language identification
Published in
arXiv.org(2018)External Links:
TL
TL;DR
This paper describes the WiLI-2018 benchmark dataset for monolingual written natural language identification, a publicly available, free of charge dataset of short text extracts from Wikipedia that contains 1000 paragraphs of 235 languages.
Abstract
This paper describes the WiLI-2018 benchmark dataset for monolingual written natural language identification. WiLI-2018 is a publicly available, free of charge dataset of short text extracts from Wikipedia. It contains 1000 paragraphs of 235 languages, totaling in 23500 paragraphs. WiLI is a classification dataset: Given an unknown paragraph written in one dominant language, it has to be decided which language it is.
References35 items
1
Analysis and Optimization of Convolutional Neural Network Architectures
2
TweetLID: a benchmark for tweet language identification
3
Adam: A Method for Stochastic Optimization
4
Automatic Detection and Language Identification of Multilingual Documents
5