Composing Text and Image for Image Retrieval - an Empirical Odyssey (2018-12-18T00:00:00.000000Z)

TL;DR

This paper proposes a new way to combine image and text through residual connection, that outperforms existing approaches on 3 different datasets, namely Fashion-200k, MIT-States and a new synthetic dataset the authors create based on CLEVR.

Abstract

In this paper, we study the task of image retrieval, where the input query is specified in the form of an image plus some text that describes desired modifications to the input image. For example, we may present an image of the Eiffel tower, and ask the system to find images which are visually similar, but are modified in small ways, such as being taken at nighttime instead of during the day. o tackle this task, we embed the query (reference image plus modification text) and the target (images). The encoding function of the image text query learns a representation, such that the similarity with the target image representation is high iff it is a ``positive match''. We propose a new way to combine image and text through residual connection, that is designed for this retrieval task. We show this outperforms existing approaches on 3 different datasets, namely Fashion-200k, MIT-States and a new synthetic dataset we create based on CLEVR. We also show that our approach can be used to perform image classification with compositionally novel labels, and we outperform previous methods on MIT-States on this task.

Authors

K. Murphy

12 papers

Chen Sun

11 papers

Lu Jiang

9 papers

TL;DR

Abstract

Authors

References56 items

Compositional Learning for Human Object Interaction

Focal Visual-Text Attention for Visual Question Answering

Learning Attribute Representations with Localization for Flexible Fashion Search

Dialog-based Interactive Image Retrieval

Dynamic Few-Shot Visual Learning Without Forgetting

Attributes as Operators

MAttNet: Modular Attention Network for Referring Expression Comprehension

Language-Based Image Editing with Recurrent Attentive Models

iVQA: Inverse Visual Question Answering

FiLM: Visual Reasoning with a General Conditioning Layer

Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge

Automatic Spatially-Aware Fashion Concept Discovery

From Red Wine to Red Tomato: Composition with Context

Memory-Augmented Attribute Manipulation Networks for Interactive Fashion Search

Learnable pooling with Context Gating for video classification

A simple neural network module for relational reasoning

In Defense of the Triplet Loss for Person Re-Identification

No Fuss Distance Metric Learning Using Proxies

Prototypical Networks for Few-shot Learning

A Joint Speaker-Listener-Reinforcer Model for Referring Expressions

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Improved Deep Metric Learning with Multi-class N-pair Loss Objective

Dual Attention Networks for Multimodal Reasoning and Matching

Localizing and Orienting Street Views Using Overhead Imagery

DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations

Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

Multimodal Residual Learning for Visual QA

CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples

Deep Image Retrieval: Learning Global Representations for Image Search

Deep Residual Learning for Image Recognition

Learning Deep Structure-Preserving Image-Text Embeddings

Image Question Answering Using Convolutional Neural Network with Dynamic Parameter Prediction

Natural Language Object Retrieval

Generation and Comprehension of Unambiguous Object Descriptions

Zero-Shot Learning via Semantic Similarity Embedding

An embarrassingly simple approach to zero-shot learning

Bridging the Ultimate Semantic Gap: A Semantic Search Engine for Internet Videos

Discovering states and transformations in image collections

Learning deep representations for ground-to-aerial geolocalization

VQA: Visual Question Answering

Highway Networks

FaceNet: A unified embedding for face recognition and clustering

Show and tell: A neural image caption generator

Inferring Analogous Attributes

Learning Fine-Grained Image Similarity with Deep Ranking

Leveraging high-level and low-level features for multimedia event detection

WhittleSearch: Image search with relative attribute feedback

Recognition using visual phrases

Learning to detect unseen object classes by between-class attribute transfer

Describing objects by their attributes

IM2GPS: estimating geographic information from a single image

Learning a similarity metric discriminatively, with application to face verification

Neighbourhood Components Analysis

Relevance feedback: a power tool for interactive content-based image retrieval

The sketchy database: learning to retrieve badly drawn bunnies

Deep Face Recognition

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names