Contrastive Representation Learning: A Framework and Review (2020-10-10T00:00:00.000000Z)

TL;DR

A general Contrastive Representation Learning framework is proposed that simplifies and unifies many different contrastive learning methods and a taxonomy for each of the components is provided in order to summarise and distinguish it from other forms of machine learning.

Abstract

Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper, we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods. We also provide a taxonomy for each of the components of contrastive learning in order to summarise it and distinguish it from other forms of machine learning. We then discuss the inductive biases which are present in any contrastive learning system and we analyse our framework under different views from various sub-fields of Machine Learning. Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented. Finally, we discuss the challenges and some of the most promising future research directions ahead.

Authors

Phúc H. Lê Khắc

1 papers

G. Healy

2 papers

A. Smeaton

3 papers

TL;DR

Abstract

Authors

References143 items

ScaleNet: An Unsupervised Representation Learning Method for Limited Information

Decoupling Representation Learning from Reinforcement Learning

Delving into Inter-Image Invariance for Unsupervised Visual Representations

Self-supervised Contrastive Video-Speech Representation Learning for Ultrasound

What Should Not Be Contrastive in Contrastive Learning

Self-Supervised Learning of Audio-Visual Objects from Video

LoCo: Local Contrastive Representation Learning

Contrastive Learning for Unpaired Image-to-Image Translation

Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases

Representation Learning with Video Deep InfoMax

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

Whitening for Self-Supervised Representation Learning

Contrastive Code Representation Learning

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training

Big Self-Supervised Models are Strong Semi-Supervised Learners

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

What makes instance discrimination good for transfer learning?

Language Agnostic Speech Embeddings for Emotion Classification

COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations

CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning

Contrastive Multi-View Representation Learning on Graphs

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

What makes for good views for contrastive learning

Prototypical Contrastive Learning of Unsupervised Representations

Probing Text Models for Common Ground with Visual Representations

Audio-Visual Instance Discrimination with Cross-Modal Agreement

Supervised Contrastive Learning

CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Watching the World Go By: Representation Learning from Unlabeled Videos

A Life of Data

Multi-modal Self-Supervision from Generalized Data Transformations

A Simple Framework for Contrastive Learning of Visual Representations

Scaling Laws for Neural Language Models

Self-Supervised Learning of Pretext-Invariant Representations

Contrastive Learning of Structured World Models

Momentum Contrast for Unsupervised Visual Representation Learning

Remote Collaborative Knowledge Discovery for Better Understanding of Self-tracking Data

Contrastive Representation Distillation

A Mutual Information Maximization Perspective of Language Representation Learning

vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations

Learning Video Representations using Contrastive Bidirectional Transformer

Video Representation Learning by Dense Predictive Coding

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization

On Mutual Information Maximization for Representation Learning

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Unsupervised State Representation Learning in Atari

Contrastive Multiview Coding

Online Object Representations with Contrastive Learning

Learning Representations by Maximizing Mutual Information Across Views

Putting An End to End-to-End: Gradient-Isolated Learning of Representations

Data-Efficient Image Recognition with Contrastive Predictive Coding

On Variational Bounds of Mutual Information

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

Unsupervised Embedding Learning via Invariant and Spreading Instance Feature

Local Aggregation for Unsupervised Learning of Visual Embeddings

A Theoretical Analysis of Contrastive Unsupervised Representation Learning

Analysing Keystroke Logging Data from a Linguistic Perspective

How Do We Sleep: a Case Study of Sleep Duration and Quality Using Data from Oura Ring

How Powerful are Graph Neural Networks?

Deep Graph Infomax

Neural Predictive Belief Representations

Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency

Learning deep representations by mutual information estimation and maximization

Learning Actionable Representations from Visual Observations

Representation Learning with Contrastive Predictive Coding

Unsupervised Feature Learning via Non-parametric Instance Discrimination

An efficient framework for learning sentence representations

Wearable sensors for clinical applications in epilepsy, Parkinson’s disease, and stroke: a mixed-methods systematic review

Objects that Sound

Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification

Children’s everyday exposure to food marketing: an objective analysis using wearable cameras

Sampling Matters in Deep Embedding Learning

Attention is All you Need