Semantic Sensitive TF-IDF to Determine Word Relevance in Documents
Published in
Lecture Notes in Electrical Engineering(2020)External Links:
TL
TL;DR
A set of nearly four million documents from health-care social media was collected and was trained in order to draw semantic model and to find the word embeddings, and the features of semantic space were utilized to rearrange the original TF-IDF scores through an iterative solution so as to improve the moderate performance of this algorithm on informal texts.
Abstract
Keyword extraction has received an increasing attention as an important research topic which can lead to have advancements in diverse applications such as document context categorization, text indexing and document classification. In this paper we propose STF-IDF, a novel semantic method based on TF-IDF, for scoring word importance of informal documents in a corpus. A set of nearly four million documents from health-care social media was collected and was trained in order to draw semantic model and to find the word embeddings. Then, the features of semantic space were utilized to rearrange the original TF-IDF scores through an iterative solution so as to improve the moderate performance of this algorithm on informal texts. After testing the proposed method with 160 randomly chosen documents, our method managed to decrease the TF-IDF mean error rate by a factor of 50% and reaching the mean error of 13.7%, as opposed to 27.2% of the original TF-IDF.
References20 items
1
An Efficient and Unique TF/IDF Algorithmic Model-Based Data Analysis for Handling Applications with Big Data Streaming
2
Multi-co-training for document classification using various document representations: TF-IDF, LDA, and Doc2Vec
3
Word Embeddings: A Survey
4
Unsupervised Sentence Representations as Word Information Series: Revisiting TF-IDF
5