High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Published in

(2017)

External Links:

Generate Graph DownloadPDF

TL;DR

A new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs) is presented, which significantly outperforms existing methods, advancing both the quality and the resolution of deep image synthesis and editing.

Abstract

We present a new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs). Conditional GANs have enabled a variety of applications, but the results are often limited to low-resolution and still far from realistic. In this work, we generate 2048 Ã— 1024 visually appealing results with a novel adversarial loss, as well as new multi-scale generator and discriminator architectures. Furthermore, we extend our framework to interactive visual manipulation with two additional features. First, we incorporate object instance segmentation information, which enables object manipulations such as removing/adding objects and changing the object category. Second, we propose a method to generate diverse results given the same input, allowing users to edit the object appearance interactively. Human opinion studies demonstrate that our method significantly outperforms existing methods, advancing both the quality and the resolution of deep image synthesis and editing.

Authors

Bryan Catanzaro

12 papers

Ming-Yu Liu

13 papers

Jan Kautz

9 papers

References69 items

End-to-End Adversarial Retinal Image Synthesis

Toward Multimodal Image-to-Image Translation

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Synthetic Medical Images from Dual Generative Adversarial Networks

Photographic Image Synthesis with Cascaded Refinement Networks

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Published in

(2017)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Bryan Catanzaro

12 papers

Ming-Yu Liu

13 papers

Jan Kautz

9 papers

References69 items

End-to-End Adversarial Retinal Image Synthesis

Toward Multimodal Image-to-Image Translation

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Synthetic Medical Images from Dual Generative Adversarial Networks

Photographic Image Synthesis with Cascaded Refinement Networks

Jun-Yan Zhu

10 papers

Ting-Chun Wang

6 papers

Scene Parsing through ADE20K Dataset

Semantic Image Synthesis via Adversarial Learning

Globally and locally consistent image completion

MoCoGAN: Decomposing Motion and Content for Video Generation

Generative Attribute Controller with Conditional Filtered Generative Adversarial Networks

Perceptual Generative Adversarial Networks for Small Object Detection

Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision

Real-time user-guided image colorization with learned deep priors

Multi-agent Diverse Generative Adversarial Networks

DualGAN: Unsupervised Dual Learning for Image-to-Image Translation

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks

Unsupervised Image-to-Image Translation Networks

Age Progression/Regression by Conditional Adversarial Autoencoder

Learning from Simulated and Unsupervised Images through Adversarial Training

Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

Stacked Generative Adversarial Networks

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

Pyramid Scene Parsing Network

Scribbler: Controlling Deep Image Synthesis with Sketch and Color

Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

Image-to-Image Translation with Conditional Adversarial Networks

Least Squares Generative Adversarial Networks

Deep joint demosaicking and denoising

Generative Multi-Adversarial Networks

Unsupervised Cross-Domain Image Generation

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Generative Visual Manipulation on the Natural Image Manifold

Energy-based Generative Adversarial Network

Generating Videos with Scene Dynamics

Instance Normalization: The Missing Ingredient for Fast Stylization

Image Style Transfer Using Convolutional Neural Networks

Coupled Generative Adversarial Networks

Improved Techniques for Training GANs

Context Encoders: Feature Learning by Inpainting

The Cityscapes Dataset for Semantic Urban Scene Understanding

Colorful Image Colorization

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Generative Image Modeling Using Style and Structure Adversarial Networks

Generating Images with Perceptual Similarity Metrics based on Deep Networks

Autoencoding beyond pixels using a learned similarity metric

Deep Residual Learning for Image Recognition

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Deep multi-scale video prediction beyond mean square error

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

U-Net: Convolutional Networks for Biomedical Image Segmentation

Adam: A Method for Stochastic Optimization

Fully convolutional networks for semantic segmentation

Conditional Generative Adversarial Nets

Very Deep Convolutional Networks for Large-Scale Image Recognition

Generative adversarial networks

Exemplar-Based Face Parsing

Interactive Facial Feature Localization

Indoor Segmentation and Support Inference from RGBD Images

Sketch2Photo: internet image montage

Semantic Photo Synthesis

Recognising panoramas

Nonlinear total variation based noise removal algorithms

The Laplacian Pyramid as a Compact Image Code

AUTO-ENCODING VARIATIONAL BAYES

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Derek Hoiem and R

Field of Study

Computer Science

Journal Information

Name

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

Page

586-595

TL;DR

Abstract

Authors

References69 items

End-to-End Adversarial Retinal Image Synthesis

Toward Multimodal Image-to-Image Translation

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Synthetic Medical Images from Dual Generative Adversarial Networks

Photographic Image Synthesis with Cascaded Refinement Networks

TL;DR

Abstract

Authors

References69 items

End-to-End Adversarial Retinal Image Synthesis

Toward Multimodal Image-to-Image Translation

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Synthetic Medical Images from Dual Generative Adversarial Networks

Photographic Image Synthesis with Cascaded Refinement Networks

Scene Parsing through ADE20K Dataset

Semantic Image Synthesis via Adversarial Learning

Globally and locally consistent image completion

MoCoGAN: Decomposing Motion and Content for Video Generation

Generative Attribute Controller with Conditional Filtered Generative Adversarial Networks

Perceptual Generative Adversarial Networks for Small Object Detection

Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision

Real-time user-guided image colorization with learned deep priors

Multi-agent Diverse Generative Adversarial Networks

DualGAN: Unsupervised Dual Learning for Image-to-Image Translation

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks

Unsupervised Image-to-Image Translation Networks

Age Progression/Regression by Conditional Adversarial Autoencoder

Wasserstein GAN

Learning from Simulated and Unsupervised Images through Adversarial Training

Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

Stacked Generative Adversarial Networks

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

Pyramid Scene Parsing Network

Scribbler: Controlling Deep Image Synthesis with Sketch and Color

Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

Image-to-Image Translation with Conditional Adversarial Networks

Least Squares Generative Adversarial Networks

Deep joint demosaicking and denoising

Generative Multi-Adversarial Networks

Unsupervised Cross-Domain Image Generation

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Generative Visual Manipulation on the Natural Image Manifold

Energy-based Generative Adversarial Network

Generating Videos with Scene Dynamics

Instance Normalization: The Missing Ingredient for Fast Stylization

Image Style Transfer Using Convolutional Neural Networks

Coupled Generative Adversarial Networks

Improved Techniques for Training GANs

Context Encoders: Feature Learning by Inpainting

The Cityscapes Dataset for Semantic Urban Scene Understanding

Colorful Image Colorization

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Generative Image Modeling Using Style and Structure Adversarial Networks

Generating Images with Perceptual Similarity Metrics based on Deep Networks

Autoencoding beyond pixels using a learned similarity metric

Deep Residual Learning for Image Recognition

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Deep multi-scale video prediction beyond mean square error

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

U-Net: Convolutional Networks for Biomedical Image Segmentation

Adam: A Method for Stochastic Optimization

Fully convolutional networks for semantic segmentation

Conditional Generative Adversarial Nets

Very Deep Convolutional Networks for Large-Scale Image Recognition

Generative adversarial networks

Exemplar-Based Face Parsing

Interactive Facial Feature Localization

Indoor Segmentation and Support Inference from RGBD Images

Sketch2Photo: internet image montage

Et al

Photo clip art

Semantic Photo Synthesis

Recognising panoramas

Nonlinear total variation based noise removal algorithms

The Laplacian Pyramid as a Compact Image Code

AUTO-ENCODING VARIATIONAL BAYES