natural-language-processing-11

Lexical Analysis

3260 papers • 126 benchmarks • 313 datasets

Lexical analysis is the process of converting a sequence of characters into a sequence of tokens (strings with an assigned and thus identified meaning). (Source: Adapted from Wikipedia)

(Image credit: Papersgraph)

Benchmarks

These leaderboards are used to track progress in lexical-analysis-11

Trend

Dataset

Best Model

Actions

No benchmarks available.

Libraries

i

Use these libraries to find lexical-analysis-11 models and implementations

Datasets

UzWordnet

Subtasks

Lexical Complexity Prediction

Most implemented papers

Chinese Lexical Analysis with Deep Bi-GRU-CRF Network

Ke Sun, Zhenyu Jiao, Shuyu Sun•Wed Jul 04 2018

A deep Bi-GRU-CRF network that jointly models word segmentation, part-of-speech tagging and named entity recognition tasks and achieves a 95.5% accuracy on the test set, roughly 13% relative error reduction over the best Chinese lexical analysis tool.

61

Content

0

Paper Graph

cs60075_team2 at SemEval-2021 Task 1 : Lexical Complexity Prediction using Transformer-based Language Models pre-trained on various text corpora

Abhilash Nandy, Sayantan Adak, Tanurima Halder, Sai Mahesh Pokala•Thu Jun 03 2021

This paper fine-tune transformer-based language models pre-trained on several text corpora, some being general (E.g., Wikipedia, BooksCorpus), some being the corpora from which the CompLex Dataset was extracted, and others being from other specific domains such as Finance, Law, etc.

6 0

Paper Graph

Monitoring term drift based on semantic consistency in an evolving vector field

Y. Kompatsiaris, P. Wittek, Sándor Darányi, Efstratios Kontopoulos, Theodoros Moysiadis•Wed Feb 04 2015

Based on the Aristotelian concept of potentiality vs. actuality allowing for the study of energy and dynamics in language, we propose a field approach to lexical analysis. Falling back on the distributional hypothesis to statistically model word meaning, we used evolving fields as a metaphor to express time-dependent changes in a vector space model by a combination of random indexing and evolving self-organizing maps (ESOM). To monitor semantic drifts within the observation period, an experiment was carried out on the term space of a collection of 12.8 million Amazon book reviews. For evaluation, the semantic consistency of ESOM term clusters was compared with their respective neighbourhoods in WordNet, and contrasted with distances among term vectors by random indexing. We found that at 0.05 level of significance, the terms in the clusters showed a high level of semantic consistency. Tracking the drift of distributional patterns in the term space across time periods, we found that consistency decreased, but not at a statistically significant level. Our method is highly scalable, with interpretations in philosophy.

14 0

Paper Graph

Rethinking Text Attribute Transfer: A Lexical Analysis

Lei Li, Hao Zhou, Jiaze Chen, Yao Fu•Sat Aug 31 2019

A lexical analysis framework, the Pivot Analysis, is proposed, to quantitatively analyze the effects of these words in text attribute classification and transfer and identifies the future requirements and challenges of this task.

20 0

Paper Graph

N-LTP: An Open-source Neural Language Technology Platform for Chinese

Ting Liu, Wanxiang Che, ylfeng•Wed Sep 23 2020

N-LTP, an open-source neural language technology platform supporting six fundamental Chinese NLP tasks: lexical analysis (Chinese word segmentation, part-of-speech tagging, and named entity recognition), syntactic parsing (dependency parsing), and semantic parsing (semantic dependency parsing and semantic role labeling).

142 0

Paper Graph

American cultural regions mapped through the lexical analysis of social media

Thomas Louf, Bruno Gonçalves, J. Ramasco, David Sánchez, J. Grieve•Mon Aug 15 2022

The approach presented here is based on the principle that cultural affiliation can be inferred from the topics that people discuss among themselves, and uncovers a manifest North–South separation, and further contiguous and non-contiguous divisions that provide a comprehensive picture of modern American cultural areas.

11 0

Paper Graph

Adding a benchmark result helps the community track progress.