Word-class embeddings for multiclass text classification (2019-11-26T00:00:00.000000Z)

TL;DR

This work proposes (supervised) word-class embeddings (WCEs), and shows that, when concatenated to (unsupervised), they substantially facilitate the training of deep-learning models in multiclass classification by topic.

Abstract

Pre-trained word embeddings encode general word semantics and lexical regularities of natural language, and have proven useful across many NLP tasks, including word sense disambiguation, machine translation, and sentiment analysis, to name a few. In supervised tasks such as multiclass text classification (the focus of this article) it seems appealing to enhance word representations with ad-hoc embeddings that encode task-specific information. We propose (supervised) word-class embeddings (WCEs), and show that, when concatenated to (unsupervised) pre-trained word embeddings, they substantially facilitate the training of deep-learning models in multiclass classification by topic. We show empirical evidence that WCEs yield a consistent improvement in multiclass classification accuracy, using six popular neural architectures and six widely used and publicly available datasets for multiclass text classification. One further advantage of this method is that it is conceptually simple and straightforward to implement. Our code that implements WCEs is publicly available at https://github.com/AlexMoreo/word-class-embeddings.

Authors

Andrea Esuli

4 papers

Alejandro Moreo

3 papers

F. Sebastiani

2 papers

TL;DR

Abstract

Authors

References103 items

Generalized Funnelling: Ensemble Learning and Heterogeneous Document Embeddings for Cross-Lingual Text Classification

Task-Optimized Word Embeddings for Text Classification Representations

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Contextual Generation of Word Embeddings for Out of Vocabulary Words in Downstream Tasks

Enhancing Domain Word Embedding via Latent Semantic Imputation

A Weighted Word Embedding Model for Text Classification

Incorporating Task-Oriented Representation in Text Classification

Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models

Learning to Weight for Text Classification

The Neural Hype and Comparisons Against Weak Baselines

GILE: A Generalized Input-Label Embedding for Text Classification

Joint Embedding of Words and Labels for Text Classification

From Word to Sense Embeddings: A Survey on Vector Representations of Meaning

Deep Contextualized Word Representations

Deep learning for sentiment analysis: A survey

Advances in Pre-Training Distributed Word Representations

A Review on Quantification Learning

Learned in Translation: Contextualized Word Vectors

Do Convolutional Networks need to be Deep for Text Classification ?

Attention is All you Need

The European Chapter of the Association for Computational Linguistics (EACL 2017)

The Effects of Data Size and Frequency Range on Distributional Semantic Models

Enriching Word Vectors with Subword Information

Bag-of-Embeddings for Text Classification

Bag of Tricks for Efficient Text Classification

Text classification based on deep belief network and softmax regression

Sparse Local Embeddings for Extreme Multi-label Classification

Semi-Automated Text Classification for Sensitivity Identification

Character-level Convolutional Networks for Text Classification

Character-Aware Neural Language Models

Effective Approaches to Attention-based Neural Machine Translation

PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks

Improving Distributional Similarity with Lessons Learned from Word Embeddings

Label-Embedding for Image Classification

Recurrent Convolutional Neural Networks for Text Classification

Adam: A Method for Stochastic Optimization

Neural Word Embedding as Implicit Matrix Factorization

GloVe: Global Vectors for Word Representation

Evaluation of output embeddings for fine-grained image classification

Convolutional Neural Networks for Sentence Classification

Multi-label Classification via Feature-aware Implicit Label Space Encoding

Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors

Zero-Shot Learning by Convex Combination of Semantic Embeddings

Learning word embeddings efficiently with noise-contrastive estimation

Distributed Representations of Words and Phrases and their Compositionality

Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Large-scale Multi-label Learning with Missing Labels

Efficient Estimation of Word Representations in Vector Space

Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Autoencoders, Unsupervised Learning, and Deep Architectures

Natural Language Processing (Almost) from Scratch

Label Embedding Trees for Large Multi-Class Tasks

Large scale image annotation: learning to rank with joint word-image embeddings

Understanding the difficulty of training deep feedforward neural networks

Why Does Unsupervised Pre-training Help Deep Learning?

Supervised and Traditional Term Weighting Methods for Automatic Text Categorization

Multi-Label Prediction via Compressed Sensing

Latent semantic analysis

Opinion Mining and Sentiment Analysis

Extracting semantic representations from word co-occurrence statistics: A computational study

Frustratingly Easy Domain Adaptation

The JRC-Acquis: A Multilingual Aligned Parallel Corpus with 20+ Languages

Domain Adaptation with Structural Correspondence Learning

A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data

An Introduction to Random Indexing

Beyond TFIDF Weighting for Text Categorization in the Vector Space Model

Exploiting Subjectivity Classification to Improve Information Extraction

A pitfall and solution in multi-class feature selection for text classification

Supervised term weighting for automated text categorization

Distributional Word Clusters vs. Words for Text Categorization

A Neural Probabilistic Language Model

A statistical learning learning model of text classification for support vector machines

Text Categorization with Support Vector Machines: Learning with Many Relevant Features

Combining Statistical Learning with a Knowledge-Based Approach - A Case Study in Intensive Care Monitoring

Distributional clustering of words for text classification

Long Short-Term Memory