TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning (2021-04-14T00:00:00.000000Z)

TL;DR

A new state-of-the-art unsupervised method based on pre-trained Transformers and Sequential Denoising Auto-Encoder (TSDAE) which outperforms previous approaches by up to 6.4 points and can achieve up to 93.1% of the performance of in-domain supervised approaches.

Abstract

Learning sentence embeddings often requires a large amount of labeled data. However, for most tasks and domains, labeled data is seldom available and creating it is expensive. In this work, we present a new state-of-the-art unsupervised method based on pre-trained Transformers and Sequential Denoising Auto-Encoder (TSDAE) which outperforms previous approaches by up to 6.4 points. It can achieve up to 93.1% of the performance of in-domain supervised approaches. Further, we show that TSDAE is a strong domain adaptation and pre-training method for sentence embeddings, significantly outperforming other approaches like Masked Language Model. A crucial shortcoming of previous studies is the narrow evaluation: Most work mainly evaluates on the single task of Semantic Textual Similarity (STS), which does not require any domain knowledge. It is unclear if these proposed methods generalize to other domains and tasks. We fill this gap and evaluate TSDAE and other recent approaches on four different datasets from heterogeneous domains.

Authors

Iryna Gurevych

3 papers

Nils Reimers

8 papers

Kexin Wang

1 papers

TL;DR

Abstract

Authors

References46 items

SimCSE: Simple Contrastive Learning of Sentence Embeddings

Whitening Sentence Representations for Better Semantics and Faster Retrieval

On the Sentence Embeddings from BERT for Semantic Textual Similarity

DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

SPECTER: Document-level Representation Learning using Citation-informed Transformers

A Simple Framework for Contrastive Learning of Visual Representations

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Multilingual Universal Sentence Encoder for Semantic Retrieval

Universal Sentence Encoder for English

Learning Universal Sentence Representations with Mean-Max Attention Autoencoder

Glow: Generative Flow with Invertible 1x1 Convolutions

A Continuously Growing Dataset of Sentential Paraphrases

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

Attention is All you Need

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Efficient Natural Language Response Suggestion for Smart Reply

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features

Task-Oriented Intrinsic Evaluation of Semantic Textual Similarity

Learning Distributed Representations of Sentences from Unlabelled Data

Semi-supervised Question Retrieval with Gated Convolutions

CQADupStack: A Benchmark Data Set for Community Question-Answering Research

A large annotated corpus for learning natural language inference

SemEval-2015 Task 1: Paraphrase and Semantic Similarity in Twitter (PIT)

SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability

GloVe: Global Vectors for Word Representation

SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

The Stanford CoreNLP Natural Language Processing Toolkit

A SICK cure for the evaluation of compositional distributional semantic models

*SEM 2013 shared task: Semantic Textual Similarity

SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion

Room for a view: North

Dimensionality Reduction by Learning an Invariant Mapping

Tension

Semantic Re-tuning with Contrastive Tension

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation

Deep Learning

GatfordCentre for Interactive Systems ResearchDepartment of Information

Amaru Cuba Gyllensten, and Erik Ylipää Hellqvist. 2021. Semantic re-tuning with contrastive

and the 9th International Joint Conference on Natural Language Processing

We use the repository of sentence-transformers 8 (version number: v0.3.8) to train the in-domain supervised models. For them, the number of training epochs is 10

Field of Study

Venue Information

Name

Type

URL

Alternate Names