AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017-11-28T00:00:00.000000Z)

TL;DR

An Attentional Generative Adversarial Network that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation and for the first time shows that the layered attentional GAN is able to automatically select the condition at the word level for generating different parts of the image.

Abstract

In this paper, we propose an Attentional Generative Adversarial Network (AttnGAN) that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation. With a novel attentional generative network, the AttnGAN can synthesize fine-grained details at different sub-regions of the image by paying attentions to the relevant words in the natural language description. In addition, a deep attentional multimodal similarity model is proposed to compute a fine-grained image-text matching loss for training the generator. The proposed AttnGAN significantly outperforms the previous state of the art, boosting the best reported inception score by 14.14% on the CUB dataset and 170.25% on the more challenging COCO dataset. A detailed analysis is also performed by visualizing the attention layers of the AttnGAN. It for the first time shows that the layered attentional GAN is able to automatically select the condition at the word level for generating different parts of the image.

Authors

Zhe Gan

19 papers

Xiaodong He

16 papers

Pengchuan Zhang

8 papers

TL;DR

Abstract

Authors

References41 items

Context Encoding for Semantic Segmentation

Densely Connected Pyramid Dehazing Network

Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network

Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network

Improving GANs Using Optimal Transport

A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts

StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks

MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network

Attention is All you Need

Multi-style Generative Network for Real-time Transfer

Parallel Multiscale Autoregressive Density Estimation

Image De-Raining Using a Conditional Generative Adversarial Network

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

Semantic Compositional Networks for Visual Captioning

Image-to-Image Translation with Conditional Adversarial Networks

Learning What and Where to Draw

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Conditional Image Generation with PixelCNN Decoders

Improved Techniques for Training GANs

Generative Adversarial Text to Image Synthesis

Learning Deep Representations of Fine-Grained Visual Descriptions

Rethinking the Inception Architecture for Computer Vision

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Generating Images from Captions with Attention

Stacked Attention Networks for Image Question Answering

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

VQA: Visual Question Answering

DRAW: A Recurrent Neural Network For Image Generation

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

From captions to visual concepts and back

Neural Machine Translation by Jointly Learning to Align and Translate

ImageNet Large Scale Visual Recognition Challenge

Microsoft COCO: Common Objects in Context

Auto-Encoding Variational Bayes

Learning deep structured semantic models for web search using clickthrough data

The Caltech-UCSD Birds-200-2011 Dataset

Discriminative learning in sequential pattern recognition

Bidirectional recurrent neural networks

Minimum classification error rate methods for speech recognition

GENERATIVE ADVERSARIAL NETS

Supplementary materials for: Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space

Field of Study

Journal Information

Name

Page