Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task (2019-10-08T00:00:00.000000Z)

TL;DR

It is shown that the proposed new approach to learn multimodal multilingual embeddings for matching images and their relevant captions in two languages enables better generalization, achieving state-of-the-art performance in text-to-image and image- to-text retrieval task, and caption-caption similarity task.

Abstract

In this paper, we propose a new approach to learn multimodal multilingual embeddings for matching images and their relevant captions in two languages. We combine two existing objective functions to make images and captions close in a joint embedding space while adapting the alignment of word embeddings between existing languages in our model. We show that our approach enables better generalization, achieving state-of-the-art performance in text-to-image and image-to-text retrieval task, and caption-caption similarity task. Two multimodal multilingual datasets are used for evaluation: Multi30k with German and English captions and Microsoft-COCO with English and Japanese captions.

Authors

K. Aberer

2 papers

R. Lebret

2 papers

Alireza Mohammadshahi

1 papers

TL;DR

Abstract

Authors

References40 items

Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion

Neural Baby Talk

Unsupervised Machine Translation Using Monolingual Corpora Only

Word Translation Without Parallel Data

Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (Extended Abstract)

Bottom-Up and Top-Down Attention for Image Captioning and VQA

Image Pivoting for Learning Multilingual Multimodal Representations

VSE++: Improved Visual-Semantic Embeddings

Attention is All you Need

STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset

Learning Two-Branch Neural Networks for Image-Text Matching Tasks

Multilingual Multi-modal Embeddings for Natural Language Processing

Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning

Self-Critical Sequence Training for Image Captioning

Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM

A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

Cross-Lingual Image Caption Generation

Multi30K: Multilingual English-German Image Descriptions

Multimodal Pivots for Image Caption Translation

Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures

Deep Residual Learning for Image Recognition

Order-Embeddings of Images and Language

Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning

Image-Mediated Learning for Zero-Shot Cross-Lingual Document Retrieval

Combining Language and Vision with a Multimodal Skip-gram Model

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Improving zero-shot learning by mitigating the hubness problem

Deep visual-semantic alignments for generating image descriptions

Long-term recurrent convolutional networks for visual recognition and description

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

Microsoft COCO: Common Objects in Context

Grounded Compositional Semantics for Finding and Describing Images with Sentences

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

DeViSE: A Deep Visual-Semantic Embedding Model

Exploiting Similarities among Languages for Machine Translation

Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

Applying Conditional Random Fields to Japanese Morphological Analysis

Europarl: A Parallel Corpus for Statistical Machine Translation

SHEEP, GOATS, LAMBS and WOLVES A Statistical Analysis of Speaker Performance in the NIST 1998 Speaker Recognition Evaluation

Field of Study

Venue Information

Name

Type

URL

Alternate Names