Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020-03-25T00:00:00.000000Z)

TL;DR

An intuitive paradigm for the task, layout-to-mask- to-image, which learns to unfold object masks in a weakly-supervised way based on an input layout and object style codes is proposed and a method built on Generative Adversarial Networks (GANs) is presented.

Abstract

With the remarkable recent progress on learning deep generative models, it becomes increasingly interesting to develop models for controllable image synthesis from reconfigurable structured inputs. This paper focuses on a recently emerged task, layout-to-image, whose goal is to learn generative models for synthesizing photo-realistic images from a spatial layout (i.e., object bounding boxes configured in an image lattice) and its style codes (i.e., structural and appearance variations encoded by latent vectors). This paper first proposes an intuitive paradigm for the task, layout-to-mask-to-image, which learns to unfold object masks in a weakly-supervised way based on an input layout and object style codes. The layout-to-mask component deeply interacts with layers in the generator network to bridge the gap between an input layout and synthesized images. Then, this paper presents a method built on Generative Adversarial Networks (GANs) for the proposed layout-to-mask-to-image synthesis with layout and style control at both image and object levels. The controllability is realized by a proposed novel Instance-Sensitive and Layout-Aware Normalization (ISLA-Norm) scheme. A layout semi-supervised version of the proposed method is further developed without sacrificing performance. In experiments, the proposed method is tested in the COCO-Stuff dataset and the Visual Genome dataset with state-of-the-art performance obtained.

TL;DR

Abstract

Authors

References59 items

Layout2image: Image Generation from Layout

PointRend: Image Segmentation As Rendering

Semantic Object Accuracy for Generative Text-to-Image Synthesis

Specifying Object Attributes and Relations in Interactive Scene Generation

Image Synthesis From Reconfigurable Layout and Style

Classification Accuracy Score for Conditional Generative Models

Few-Shot Unsupervised Image-to-Image Translation

PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph

Semantic Image Synthesis With Spatially-Adaptive Normalization

Object-Driven Text-To-Image Synthesis via Adversarial Training

Generating Multiple Objects at Spatially Distinct Locations

Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model

A Style-Based Generator Architecture for Generative Adversarial Networks

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Self-Attention Generative Adversarial Networks

Multimodal Unsupervised Image-to-Image Translation

Image Generation from Scene Graphs

Spectral Normalization for Generative Adversarial Networks

cGANs with Projection Discriminator

Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

Toward Multimodal Image-to-Image Translation

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Wasserstein Generative Adversarial Networks

Modulating early visual processing by language

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Geometric GAN

Mask R-CNN

Learning to Discover Cross-Domain Relations with Generative Adversarial Networks

Hierarchical Implicit Models and Likelihood-Free Variational Inference

COCO-Stuff: Thing and Stuff Classes in Context

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

Feature Pyramid Networks for Object Detection

Image-to-Image Translation with Conditional Adversarial Networks

Conditional Image Synthesis with Auxiliary Classifier GANs

A Learned Representation For Artistic Style

Variational Autoencoder for Deep Learning of Images, Labels and Captions

Conditional Image Generation with PixelCNN Decoders

Improved Techniques for Training GANs

Adversarially Learned Inference

Generative Adversarial Text to Image Synthesis

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Pixel Recurrent Neural Networks

Deep Residual Learning for Image Recognition

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Generating Images from Captions with Attention

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

Conditional Generative Adversarial Nets

Very Deep Convolutional Networks for Large-Scale Image Recognition

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Auto-Encoding Variational Bayes

GENERATIVE ADVERSARIAL NETS

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

The Frkhet Distance between Multivariate Normal Distributions

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names