SemGloVe: Semantic Co-Occurrences for GloVe From BERT
TL
TL;DR
SemGloVe is proposed, which distills semantic co-occurrences from BERT into static GloVe word embeddings and can define the co- Occurrence weights by directly considering the semantic distance between word pairs.
Abstract
GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. However, word pairs in the matrices are extracted from a predefined local context window, which might lead to limited word pairs and potentially semantic irrelevant word pairs. In this paper, we propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings. Particularly, we propose two models to extract co-occurrence statistics based on either the masked language model or the multi-head attention weights of BERT. Our methods can extract word pairs limited by the local window assumption, and can define the co-occurrence weights by directly considering the semantic distance between word pairs. Experiments on several word similarity datasets and external tasks show that SemGloVe can outperform GloVe.
References55 items
1
Triggerless Backdoor Attack for NLP Tasks with Clean Labels
2
Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration
3
Obtaining Better Static Word Embeddings Using Contextual Embedding Models
4
Combining BERT with Static Word Embeddings for Categorizing Social Media
5