MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models

Published in

Conference on Computational Natural Language Le...(2021)

External Links:

Generate Graph DownloadPDF

TL;DR

This paper proposes a fully unsupervised approach to improving word-in-context (WiC) representations in PLMs, achieved via a simple and efficient WiC-targeted fine-tuning procedure: MirrorWiC.

Abstract

Recent work indicated that pretrained language models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via simple self-supervised techniques. Inspired by this line of work, in this paper we propose a fully unsupervised approach to improving word-in-context (WiC) representations in PLMs, achieved via a simple and efficient WiC-targeted fine-tuning procedure: MirrorWiC. The proposed method leverages only raw texts sampled from Wikipedia, assuming no sense-annotated data, and learns context-aware word representations within a standard contrastive learning setup. We experiment with a series of standard and comprehensive WiC benchmarks across multiple languages. Our proposed fully unsupervised MirrorWiC models obtain substantial gains over off-the-shelf PLMs across all monolingual, multilingual and cross-lingual setups. Moreover, on some standard WiC benchmarks, MirrorWiC is even on-par with supervised models fine-tuned with in-task data and sense labels.

Authors

A. Korhonen

17 papers

Ivan Vulic

27 papers

Nigel Collier

12 papers

References47 items

Self-Guided Contrastive Learning for BERT Sentence Representations

A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space

ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

Let’s Play Mono-Poly: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

SimCSE: Simple Contrastive Learning of Sentence Embeddings

MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models

Published in

Conference on Computational Natural Language Le...(2021)

External Links:

Generate Graph DownloadPDF

TL;DR

This paper proposes a fully unsupervised approach to improving word-in-context (WiC) representations in PLMs, achieved via a simple and efficient WiC-targeted fine-tuning procedure: MirrorWiC.

Abstract

Authors

A. Korhonen

17 papers

Ivan Vulic

27 papers

Nigel Collier

12 papers

References47 items

Self-Guided Contrastive Learning for BERT Sentence Representations

A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space

ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

Let’s Play Mono-Poly: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

SimCSE: Simple Contrastive Learning of Sentence Embeddings

Qianchu Liu

2 papers

Fangyu Liu

15 papers

AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examples

Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders

Don’t Invite BERT to Drink a Bottle: Modeling the Interpretation of Metonymies Using BERT and Distributional Representations

Towards Better Context-aware Lexical Semantics: Adjusting Contextualized Representations through Static Anchors

Self-Alignment Pretraining for Biomedical Entity Representations

XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization

Probing Pretrained Language Models for Lexical Semantics

Language-agnostic BERT Sentence Embedding

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders

WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context

Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

CoSimLex: A Resource for Evaluating Graded Word Similarity in Context

Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations

How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings

Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings

SenseBERT: Driving Some Sense into BERT

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation

Visualizing and Measuring the Geometry of BERT

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A Comparison of Context-sensitive Models for Lexical Substitution

WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations

Representation Learning with Contrastive Predictive Coding

From Word to Sense Embeddings: A Survey on Vector Representations of Meaning

Decoupled Weight Decay Regularization

Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison

All-but-the-Top: Simple and Effective Postprocessing for Word Representations

Nasari: Integrating explicit knowledge and corpus statistics for a multilingual representation of concepts and entities

A Latent Variable Model Approach to PMI-based Word Embeddings

A Unified Model for Word Sense Representation and Disambiguation

Measuring Word Meaning in Context

Word sense disambiguation: A survey

Semantic Re-tuning with Contrastive Tension

Bootstrapped Unsupervised Sentence Representation Learning

work of Liu et al. (

Recent advances in language model ﬁne-tuning

What do you mean, BERT?

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

We also tested with ﬁne-tuning with strictly ‘in-domain’ data, i.e., raw sentences (w/o labels) sampled from the training sets of WiC tasks, but found no substantial difference

corpora, we individually sampled ﬁve sets of 10k raw sentences and found only minor difference when ﬁne-tuning on them ( ≈ 0 . 003 standard deviation for BERT +M IRROR W I C and ≈ 0 . 001 for

Field of Study

Computer Science

Venue Information

Name

Conference on Computational Natural Language Learning

Type

conference

URL

N/A

Alternate Names

CoNLL
Conf Comput Nat Lang Learn