Lessons Learned in Multilingual Grounded Language Learning (2018-09-01T00:00:00.000000Z)

TL;DR

It is demonstrated that a multilingual model can be trained equally well on either translations or comparable sentence pairs, and that annotating the same set of images in multiple language enables further improvements via an additional caption-caption ranking objective.

Abstract

Recent work has shown how to learn better visual-semantic embeddings by leveraging image descriptions in more than one language. Here, we investigate in detail which conditions affect the performance of this type of grounded language learning model. We show that multilingual training improves over bilingual training, and that low-resource languages benefit from training with higher-resource languages. We demonstrate that a multilingual model can be trained equally well on either translations or comparable sentence pairs, and that annotating the same set of images in multiple language enables further improvements via an additional caption-caption ranking objective.

Authors

Marc-Alexandre Côté

10 papers

Desmond Elliott

8 papers

Ákos Kádár

5 papers

TL;DR

Abstract

Authors

References57 items

Visual Question Answering: From Theory to Application

Assessing multilingual multimodal image description: Studies of native speaker preferences and translator choices

Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech

Improving Visually Grounded Sentence Representations with Self-Attention

Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description

Massively Multilingual Neural Grapheme-to-Phoneme Conversion

Learning Language Representations for Typology Prediction

Image Pivoting for Learning Multilingual Multimodal Representations

Learning Visually Grounded Sentence Representations

VSE++: Improved Visual-Semantic Embeddings

VSE++: Improving Visual-Semantic Embeddings with Hard Negatives

Imagination Improves Multimodal Translation

Attention Strategies for Multi-Source Sequence-to-Sequence Learning

Multilingual Multi-modal Embeddings for Natural Language Processing

Continuous multilinguality with language vectors

Zero-resource machine translation by multimodal encoder–decoder network with multimedia pivot

Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Fully Character-Level Neural Machine Translation without Explicit Segmentation

A Shared Task on Multimodal Machine Translation and Crosslingual Image Description

Cross-Lingual Image Caption Generation

Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Adding Chinese Captions to Images

Generative Adversarial Text to Image Synthesis

Multi30K: Multilingual English-German Image Descriptions

Transfer Learning for Low-Resource Neural Machine Translation

A Character-level Decoder without Explicit Segmentation for Neural Machine Translation

Many Languages, One Parser

Deep Residual Learning for Image Recognition

Order-Embeddings of Images and Language

Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning

Image-Mediated Learning for Zero-Shot Cross-Lingual Document Retrieval

Skip-Thought Vectors

SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability

VQA: Visual Question Answering

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Universal Dependencies 1.4

Adam: A Method for Stochastic Optimization

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Deep visual-semantic alignments for generating image descriptions

Show and tell: A neural image caption generator

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics

Code Mixing: A Challenge for Language Identification in the Language of Social Media

SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

Improving Multi-Modal Representations Using Image Dispersion: Why Less is Sometimes More

A SICK cure for the evaluation of compositional distributional semantic models

ImageNet: A large-scale hierarchical image database

Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources

Grounding conceptual knowledge in modality-specific systems

Grounding Distributional Semantics in the Visual World

Dataset of images and sentence descriptions Inferred correspondences Generative model training image " A Tabby cat is leaning on a wooden table , with one paw on a laser mouse and the other on a black laptop " training image

Learning word meanings from images of natural scenes

Multimodal Distributional Semantics

Placing search in context: the concept revisited

Additional languages. Multilingual models have not been explored yet in a multimodal setting

Field of Study

Venue Information

Name

Type

URL

Alternate Names