Multi-modal Cycle-consistent Generalized Zero-Shot Learning (2018-08-01T00:00:00.000000Z)

TL;DR

This paper proposes the use of a constraint based on a new regularization for the GAN training that forces the generated visual features to reconstruct their original semantic features in a multi-modal cycle-consistent manner.

Abstract

In generalized zero shot learning (GZSL), the set of classes are split into seen and unseen classes, where training relies on the semantic features of the seen and unseen classes and the visual representations of only the seen classes, while testing uses the visual representations of the seen and unseen classes. Current methods address GZSL by learning a transformation from the visual to the semantic space, exploring the assumption that the distribution of classes in the semantic and visual spaces is relatively similar. Such methods tend to transform unseen testing visual representations into one of the seen classes’ semantic features instead of the semantic features of the correct unseen class, resulting in low accuracy GZSL classification. Recently, generative adversarial networks (GAN) have been explored to synthesize visual representations of the unseen classes from their semantic features - the synthesized representations of the seen and unseen classes are then used to train the GZSL classifier. This approach has been shown to boost GZSL classification accuracy, but there is one important missing constraint: there is no guarantee that synthetic visual representations can generate back their semantic feature in a multi-modal cycle-consistent manner. This missing constraint can result in synthetic visual representations that do not represent well their semantic features, which means that the use of this constraint can improve GAN-based approaches. In this paper, we propose the use of such constraint based on a new regularization for the GAN training that forces the generated visual features to reconstruct their original semantic features. Once our model is trained with this multi-modal cycle-consistent semantic compatibility, we can then synthesize more representative visual representations for the seen and, more importantly, for the unseen classes. Our proposed approach shows the best GZSL classification results in the field in several publicly available datasets.

References37 items

Zero-Shot Learning Using Synthesised Unseen Visual Data with Diffusion Regularisation

Preserving Semantic Relations for Zero-Shot Learning

Zero-Shot Visual Recognition Using Semantics-Preserving Adversarial Embedding Networks

Feature Generating Networks for Zero-Shot Learning

A Bayesian Data Augmentation Approach for Learning Deep Models

Generating Visual Representations for Zero-Shot Classification

Semantic Autoencoder for Zero-Shot Learning

Zero-Shot Learning — The Good, the Bad and the Ugly

Wasserstein GAN

Learning Deep Representations of Fine-Grained Visual Descriptions

Less is More: Zero-Shot Learning from Online Textual Documents with Noise Suppression

Latent Embeddings for Zero-Shot Classification

Synthesized Classifiers for Zero-Shot Learning

Deep Residual Learning for Image Recognition

Attribute2Image: Conditional Image Generation from Visual Attributes

Zero-Shot Learning via Semantic Similarity Embedding

An embarrassingly simple approach to zero-shot learning

Label-Embedding for Image Classification

Evaluation of output embeddings for fine-grained image classification

Zero-Shot Learning by Convex Combination of Semantic Embeddings

DeViSE: A Deep Visual-Semantic Embedding Model

Distributed Representations of Words and Phrases and their Compositionality

Designing Category-Level Attributes for Discriminative Visual Recognition

Zero-Shot Learning Through Cross-Modal Transfer

Caltech-UCSD Birds 200

Rectified Linear Units Improve Restricted Boltzmann Machines

SUN database: Large-scale scene recognition from abbey to zoo

Describing objects by their attributes

ImageNet: A large-scale hierarchical image database

Learning to detect unseen object classes by between-class attribute transfer

Automated Flower Classification over a Large Number of Classes

Multi-attention Network for One Shot Learning

Unpaired image-to-image translation using cycle-consistent adversarial networks

Rectifier Nonlinearities Improve Neural Network Acoustic Models

Attribute-Based Classification for Zero-Shot Visual Object Categorization

This Paper Is Included in the Proceedings of the 12th Usenix Symposium on Operating Systems Design and Implementation (osdi '16). Tensorflow: a System for Large-scale Machine Learning Tensorflow: a System for Large-scale Machine Learning

Multi-modal Cycle-consistent Generalized Zero-Shot