Picture it in your mind: generating high level visual representations from textual descriptions (2016-06-23T00:00:00.000000Z)

TL;DR

This paper proposes various neural network models of increasing complexity that learn to generate, from a short descriptive text, a high level visual representation in a visual feature space such as the pool5 layer of the ResNet-152 or the fc6–fc7 layers of an AlexNet trained on ILSVRC12 and Places databases.

Abstract

In this paper we tackle the problem of image search when the query is a short textual description of the image the user is looking for. We choose to implement the actual search process as a similarity search in a visual feature space, by learning to translate a textual query into a visual representation. Searching in the visual feature space has the advantage that any update to the translation model does not require to reprocess the (typically huge) image collection on which the search is performed. We propose various neural network models of increasing complexity that learn to generate, from a short descriptive text, a high level visual representation in a visual feature space such as the pool5 layer of the ResNet-152 or the fc6–fc7 layers of an AlexNet trained on ILSVRC12 and Places databases. The Text2Vis models we explore include (1) a relatively simple regressor network relying on a bag-of-words representation for the textual descriptors, (2) a deep recurrent network that is sensible to word order, and (3) a wide and deep model that combines a stacked LSTM deep network with a wide regressor network. We compare the models we propose with other search strategies, also including textual search methods that exploit state-of-the-art caption generation models to index the image collection.

Authors

Andrea Esuli

4 papers

Alejandro Moreo

3 papers

F. Carrara

2 papers

TL;DR

Abstract

Authors

References49 items

Wide & Deep Learning for Recommender Systems

Word2VisualVec: Cross-Media Retrieval by Visual Feature Prediction

Deep Image Retrieval: Learning Global Representations for Image Search

Deep Residual Learning for Image Recognition

Learning Deep Structure-Preserving Image-Text Embeddings

Image2Emoji: Zero-shot Emoji Prediction for Visual Media

Skip-Thought Vectors

Multimodal Convolutional Neural Networks for Matching Image and Sentence

Microsoft COCO Captions: Data Collection and Evaluation Server

YFCC100M

Adam: A Method for Stochastic Optimization

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Learning Deep Features for Scene Recognition using Places Database

Deep visual-semantic alignments for generating image descriptions

Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation

From captions to visual concepts and back

Show and tell: A neural image caption generator

Long-term recurrent convolutional networks for visual recognition and description

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Bag-of-Words Based Deep Neural Network for Image Retrieval

Cross-modal Retrieval with Correspondence Autoencoder

GloVe: Global Vectors for Word Representation

Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Microsoft COCO: Common Objects in Context

CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

Multi-scale Orderless Pooling of Deep Convolutional Activation Features

On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval

Zero-Shot Learning by Convex Combination of Semantic Embeddings

DeViSE: A Deep Visual-Semantic Embedding Model

Clickage: towards bridging semantic and intent gaps via mining click logs of search engines

Distributed Representations of Words and Phrases and their Compositionality

DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

ImageNet classification with deep convolutional neural networks

Negative Evidences and Co-occurences in Image Retrieval: The Benefit of PCA and Whitening

Multimodal Deep Learning

Et al

Computing Information Retrieval Performance Measures Efficiently in the Presence of Tied Scores

ROUGE: A Package for Automatic Evaluation of Summaries

Cumulated gain-based evaluation of IR techniques

Long Short-Term Memory

Independent component analysis, A new concept?

Backpropagation Through Time: What It Does and How to Do It

Long short-term memory recurrent neural network architectures for large scale acoustic modeling

LSTM Neural Networks for Language Modeling

Negative evidences and co-occurrences in image retrieval: the benet of PCA and whitening

Generating High Level Visual Representations From Textual Descriptions

Field of Study

Journal Information

Name

Page

Volume