natural-language-processing-2

Language Identification

3260 papers • 126 benchmarks • 313 datasets

Language identification is the task of determining the language of a text.

(Image credit: Papersgraph)

Benchmarks

These leaderboards are used to track progress in language-identification-4

Trend

Dataset

Best Model

Actions

VOXLINGUA107

OpenSubtitles

Universal Dependencies

Libraries

i

Use these libraries to find language-identification-4 models and implementations

facebookresearch/fairseq

2 papers 30,141

Datasets

Universal Dependencies

Subtasks

Dialect Identification Native Language Identification

Most implemented papers

The WiLI benchmark dataset for written language identification

Martin Thoma•Mon Jan 22 2018

This paper describes the WiLI-2018 benchmark dataset for monolingual written natural language identification, a publicly available, free of charge dataset of short text extracts from Wikipedia that contains 1000 paragraphs of 235 languages.

17

Content

Universal Dependencies

Nordic Language Identification

GlotLID-C

VoxForge

pytorch/fairseq

2 papers 30,141

Dakshina

0

SpeechBrain: A General-Purpose Speech Toolkit

Yoshua Bengio, M. Ravanelli, Samuele Cornell, Jianyuan Zhong, Cem Subakan, Szu-Wei Fu, Loren Lugosch, A. Heba, Titouan Parcollet, Ju-Chieh Chou, Peter William VanHarn Plantinga, Aku Rouhe, Nauman Dawalatabad, Sung-Lin Yeh, Chien-Feng Liao, Elena Rastorgueva, Franccois Grondin, William Aris, Hwidong Na, Yan Gao, R. Mori•Mon Jun 07 2021

The core architecture of SpeechBrain is described, designed to support several tasks of common interest, allowing users to naturally conceive, compare and share novel speech processing pipelines.

916 0

Paper Graph

GlotLID: Language Identification for Low-Resource Languages

Hinrich Schütze, Amir Hossein Kargaran, Ayyoob Imani, François Yvon•Mon Oct 23 2023

GlotLID-M is published, an LID model that satisfies the desiderata of wide coverage, reliability and efficiency, and it identifies 1665 languages, a large increase in coverage compared to prior work.

29 0

Paper Graph

Universal Dependency Parsing for Hindi-English Code-Switching

Manish Shrivastava, Irshad Ahmad Bhat, Riyaz Ahmad Bhat, D. Sharma•Sat Mar 31 2018

This paper presents a treebank of Hindi-English code-switching tweets under Universal Dependencies scheme and proposes a neural stacking model for parsing that efficiently leverages the part-of-speech tag and syntactic tree annotations in the code- Switching treebank and the preexisting Hindi and English treebanks.

66 0

Paper Graph

Predicting the Type and Target of Offensive Posts in Social Media

Sara Rosenthal, Marcos Zampieri, Preslav Nakov, S. Malmasi, N. Farra, Ritesh Kumar•Thu Jan 31 2019

The Offensive Language Identification Dataset (OLID), a new dataset with tweets annotated for offensive content using a fine-grained three-layer annotation scheme, is complied and made publicly available.

856 0

Paper Graph

SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)

Sara Rosenthal, Marcos Zampieri, Preslav Nakov, S. Malmasi, N. Farra, Ritesh Kumar•Mon Mar 18 2019

The results and the main findings of SemEval-2019 Task 6 on Identifying and Categorizing Offensive Language in Social Media (OffensEval), based on a new dataset, contain over 14,000 English tweets, are presented.

852 0

Paper Graph

Word-Level Embeddings for Cross-Task Transfer Learning in Speech Processing

M. Kegler, P. Beckmann, M. Cernak•Mon Oct 21 2019

This work introduces an encoder capturing word-level representations of speech for cross-task transfer learning and shows that the speech representation captured by the encoder through the pre-training is transferable across distinct speech processing tasks and datasets.

5 0

Paper Graph

Common Voice: A Massively-Multilingual Speech Corpus

Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, M. Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, Gregor Weber•Thu Dec 12 2019

This work presents speech recognition experiments using Mozilla’s DeepSpeech Speech-to-Text toolkit, and finds an average Character Error Rate improvement for twelve target languages, for most of these languages, these are the first ever published results on end- to-end Automatic Speech Recognition.

2106 0

Paper Graph

VOXLINGUA107: A Dataset for Spoken Language Recognition

Jörgen Valk, Tanel Alumäe•Tue Nov 24 2020

This paper generates semi-random search phrases from language-specific Wikipedia data that are then used to retrieve videos from YouTube for 107 languages and uses the data to build language recognition models for several spoken language identification tasks.

204 0

Paper Graph

Adding a benchmark result helps the community track progress.