Probabilistic Embeddings for Cross-Modal Retrieval (2021-01-13T00:00:00.000000Z)

TL;DR

It is argued that deterministic functions are not sufficiently powerful to capture one-to-many correspondences and proposed Probabilistic Cross-Modal Embedding (PCME), where samples from the different modalities are represented as probabilistic distributions in the common embedding space.

Abstract

Cross-modal retrieval methods build a common representation space for samples from multiple modalities, typically from the vision and the language domains. For images and their captions, the multiplicity of the correspondences makes the task particularly challenging. Given an image (respectively a caption), there are multiple captions (respectively images) that equally make sense. In this paper, we argue that deterministic functions are not sufficiently powerful to capture such one-to-many correspondences. Instead, we propose to use Probabilistic Cross-Modal Embedding (PCME), where samples from the different modalities are represented as probabilistic distributions in the common embedding space. Since common benchmarks such as COCO suffer from non-exhaustive annotations for cross-modal matches, we propose to additionally evaluate retrieval on the CUB dataset, a smaller yet clean database where all possible image-caption pairs are annotated. We extensively ablate PCME and demonstrate that it not only improves the retrieval performance over its deterministic counterpart but also provides uncertainty estimates that render the embeddings more interpretable. Code is available at https://github.com/naver-ai/pcme.

Authors

Yannis Kalantidis

9 papers

Sanghyuk Chun

8 papers

Seong Joon Oh

7 papers

TL;DR

Abstract

Authors

References63 items

On Semantic Similarity in Video Retrieval

Preserving Semantic Neighborhoods for Robust Cross-modal Retrieval

AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights

Slowing Down the Weight Norm Increase in Momentum-based Optimizers

Multi-Modality Cross Attention Network for Image and Sentence Matching

Context-Aware Attention Network for Image-Text Retrieval

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

Probabilistic embeddings for speaker diarization

Data Uncertainty Learning in Face Recognition

A Metric Learning Reality Check

Adaptive Offline Quintuplet Loss for Image-Text Matching

12-in-1: Multi-Task Vision and Language Representation Learning

View-Invariant Probabilistic Embedding for Human Pose

Stochastic Prototype Embeddings

Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching

CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval

Visual Semantic Reasoning for Image-Text Matching

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Probabilistic Face Embeddings

Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders

NSML: Meet the MLaaS platform with a real-world case study

Smoothing the Geometry of Probabilistic Box Embeddings

Modeling Uncertainty with Hedged Instance Embedding

Finding Beans in Burgers: Deep Semantic-Visual Embedding with Localization

Stacked Cross Attention for Image-Text Matching

Separating Self-Expression and Visual Content in Hashtag Supervision

Improved Regularization of Convolutional Neural Networks with Cutout

VSE++: Improving Visual-Semantic Embeddings with Hard Negatives

Attention is All you Need

A Mixture Model for Learning Multi-Sense Word Embeddings

Zero-Shot Learning — The Good, the Bad and the Ugly

Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM

Dual Attention Networks for Multimodal Reasoning and Matching

Semi-Supervised Classification with Graph Convolutional Networks

SGDR: Stochastic Gradient Descent with Warm Restarts

Layer Normalization

Learning Deep Representations of Fine-Grained Visual Descriptions

Deep Residual Learning for Image Recognition

Learning Deep Structure-Preserving Image-Text Embeddings

Generating Sentences from a Continuous Space

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Microsoft COCO Captions: Data Collection and Evaluation Server

FaceNet: A unified embedding for face recognition and clustering

Word Representations via Gaussian Embedding

Deep visual-semantic alignments for generating image descriptions

Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation

GloVe: Global Vectors for Word Representation

Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

Microsoft COCO: Common Objects in Context

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

Auto-Encoding Variational Bayes

DeViSE: A Deep Visual-Semantic Embedding Model

Caltech-UCSD Birds 200

Dimensionality Reduction by Learning an Invariant Mapping

Canonical Correlation Analysis: An Overview with Application to Learning Methods

Probability Product Kernels

Machine learning - a probabilistic perspective

On a measure of divergence between two statistical populations defined by their probability distributions

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names