Does Multimodality Help Human and Machine for Translation and Image Captioning? (2016-05-30T00:00:00.000000Z)

TL;DR

The systems developed by LIUM and CVC for the WMT16 Multimodal Machine Translation challenge are presented, namely phrase-based systems and attentional recurrent neural networks models trained using monomodal or multimodal data.

Abstract

This paper presents the systems developed by LIUM and CVC for the WMT16 Multimodal Machine Translation challenge. We explored various comparative methods, namely phrase-based systems and attentional recurrent neural networks models trained using monomodal or multimodal data. We also performed a human evaluation in order to estimate the usefulness of multimodal data for human machine translation and image description generation. Our systems obtained the best results for both tasks according to the automatic evaluation metrics BLEU and METEOR.

Authors

Joost van de Weijer

11 papers

Loïc Barrault

10 papers

Fethi Bougares

4 papers

TL;DR

Abstract

Authors

References31 items

Theano: A Python framework for fast computation of mathematical expressions

Multi30K: Multilingual English-German Image Descriptions

Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism

Deep Residual Learning for Image Recognition

Multi-Language Image Description with Neural Sequence Models

Multilingual Image Description with Neural Sequence Models

A Joint Dependency Model of Morphological and Syntactic Structure for Statistical Machine Translation

Findings of the 2015 Workshop on Statistical Machine Translation

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Sequence to Sequence Learning with Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Neural Machine Translation by Jointly Learning to Align and Translate

Multimodal Neural Language Models

Distributed Representations of Sentences and Documents

Recurrent Continuous Translation Models

ImageNet classification with deep convolutional neural networks

KenLM: Faster and Smaller Language Model Queries

SUN database: Large-scale scene recognition from abbey to zoo

Understanding the difficulty of training deep feedforward neural networks

METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

Moses: Open Source Toolkit for Statistical Machine Translation

Continuous Space Language Models for Statistical Machine Translation

Minimum Error Rate Training in Statistical Machine Translation

A Systematic Comparison of Various Statistical Alignment Models

SRILM - an extensible language modeling toolkit

Bleu: a Method for Automatic Evaluation of Machine Translation

Improving continuous space language models auxiliary features

Recurrent neural network based language model

Conference on Artificial Intelligence and Statistics (AISTATS'10). Society for Artificial Intelligence and Statistics

Field of Study

Venue Information

Name

Type

URL

Alternate Names