Image-to-Image Translation with Conditional Adversarial Networks

Published in

Computer Vision and Pattern Recognition(2016)

External Links:

Generate Graph DownloadPDF

TL;DR

Conditional adversarial networks are investigated as a general-purpose solution to image-to-image translation problems and it is demonstrated that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks.

Abstract

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Moreover, since the release of the pix2pix software associated with this paper, hundreds of twitter users have posted their own artistic experiments using our system. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without handengineering our loss functions either.

Authors

Phillip Isola

11 papers

Alexei A. Efros

23 papers

Jun-Yan Zhu

10 papers

References70 items

Generating Interpretable Images with Controllable Structure

Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

Learning What and Where to Draw

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Generative Visual Manipulation on the Natural Image Manifold

Image-to-Image Translation with Conditional Adversarial Networks

Published in

Computer Vision and Pattern Recognition(2016)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Phillip Isola

11 papers

Alexei A. Efros

23 papers

Jun-Yan Zhu

10 papers

References70 items

Generating Interpretable Images with Controllable Structure

Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

Learning What and Where to Draw

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Generative Visual Manipulation on the Natural Image Manifold

Tinghui Zhou

3 papers

Energy-based Generative Adversarial Network

Learning Temporal Transformations from Time-Lapse Videos

Instance Normalization: The Missing Ingredient for Fast Stylization

Image Style Transfer Using Convolutional Neural Networks

Improved Techniques for Training GANs

Generative Adversarial Text to Image Synthesis

Context Encoders: Feature Learning by Inpainting

Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks

The Cityscapes Dataset for Semantic Urban Scene Understanding

Colorful Image Colorization

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Pixel-Level Domain Transfer

Learning Representations for Automatic Colorization

Generative Image Modeling Using Style and Structure Adversarial Networks

Generating Images with Perceptual Similarity Metrics based on Deep Networks

Autoencoding beyond pixels using a learned similarity metric

Visually Indicated Sounds

Top-Down Learning for Structured Labeling with Convolutional Pseudoprior

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Deep multi-scale video prediction beyond mean square error

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

Multispectral pedestrian detection: Benchmark dataset and baseline

Texture Synthesis Using Convolutional Neural Networks

Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks

U-Net: Convolutional Networks for Biomedical Image Segmentation

Holistically-Nested Edge Detection

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Adam: A Method for Stochastic Optimization

Learning to generate chairs with convolutional neural networks

Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture

Fully convolutional networks for semantic segmentation

Conditional Generative Adversarial Nets

ImageNet Large Scale Visual Recognition Challenge

Transient attributes for high-level understanding and editing of outdoor scenes

Fine-Grained Visual Comparisons with Local Learning

Data-driven hallucination of different times of day from a single outdoor photo

Spatial Pattern Templates for Recognition of Objects with Regular Structure

What makes Paris look like Paris?

How do humans sketch objects?

Random Search for Hyper-Parameter Optimization

Selecting Receptive Fields in Deep Networks

Sketch2Photo: internet image montage

ImageNet: A large-scale hierarchical image database

A non-local algorithm for image denoising

Image quality assessment: from error visibility to structural similarity

Color Transfer between Images

Image quilting for texture synthesis and transfer

Texture synthesis by non-parametric sampling

GENERATIVE ADVERSARIAL NETS

Image Style Transfer using Convolutional Neural Networks

kenter, rolf jagerman, and daan gosman. https://dekennisvannu.nl/site/artikel/ Help-ons-kunstmatige-intelligentie-testen/ 9163

But they are largely independent of each other when the data is high-dimensional

Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification

Conditional generative adversarial nets for convolutional face generation

Learning Feature Representations with K-Means

Removing Camera Shake from a Single Photograph

Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks

Log-likelihood, Parzen window estimates, and visual fidelity of generated samples

Evaluating Generative Models • How to measure model qualities?

• Training is done by backpropagation but difficult and usually unstable

Bertrand gondouin

Field of Study

Computer Science

Journal Information

Name

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Page

1800-1807

Venue Information

Name

Computer Vision and Pattern Recognition

Type

conference

URL

https://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000147

Alternate Names

CVPR
Comput Vis Pattern Recognit

TL;DR

Abstract

Authors

References70 items

Generating Interpretable Images with Controllable Structure

Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

Learning What and Where to Draw

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Generative Visual Manipulation on the Natural Image Manifold

TL;DR

Abstract

Authors

References70 items

Generating Interpretable Images with Controllable Structure

Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts

Learning What and Where to Draw

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Generative Visual Manipulation on the Natural Image Manifold

Energy-based Generative Adversarial Network

Learning Temporal Transformations from Time-Lapse Videos

Instance Normalization: The Missing Ingredient for Fast Stylization

Image Style Transfer Using Convolutional Neural Networks

Improved Techniques for Training GANs

Generative Adversarial Text to Image Synthesis

Context Encoders: Feature Learning by Inpainting

Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks

The Cityscapes Dataset for Semantic Urban Scene Understanding

Colorful Image Colorization

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Pixel-Level Domain Transfer

Learning Representations for Automatic Colorization

Generative Image Modeling Using Style and Structure Adversarial Networks

Generating Images with Perceptual Similarity Metrics based on Deep Networks

Autoencoding beyond pixels using a learned similarity metric

Visually Indicated Sounds

Top-Down Learning for Structured Labeling with Convolutional Pseudoprior

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Deep multi-scale video prediction beyond mean square error

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

Multispectral pedestrian detection: Benchmark dataset and baseline

Texture Synthesis Using Convolutional Neural Networks

Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks

U-Net: Convolutional Networks for Biomedical Image Segmentation

Holistically-Nested Edge Detection

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Adam: A Method for Stochastic Optimization

Learning to generate chairs with convolutional neural networks

Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture

Fully convolutional networks for semantic segmentation

Conditional Generative Adversarial Nets

ImageNet Large Scale Visual Recognition Challenge

Transient attributes for high-level understanding and editing of outdoor scenes

Fine-Grained Visual Comparisons with Local Learning

Data-driven hallucination of different times of day from a single outdoor photo

Spatial Pattern Templates for Recognition of Objects with Regular Structure

What makes Paris look like Paris?

How do humans sketch objects?

Random Search for Hyper-Parameter Optimization

Selecting Receptive Fields in Deep Networks

Sketch2Photo: internet image montage

ImageNet: A large-scale hierarchical image database

A non-local algorithm for image denoising

Image quality assessment: from error visibility to structural similarity

Color Transfer between Images

Image quilting for texture synthesis and transfer

Image Analogies

Texture synthesis by non-parametric sampling

GENERATIVE ADVERSARIAL NETS

Image Style Transfer using Convolutional Neural Networks

kenter, rolf jagerman, and daan gosman. https://dekennisvannu.nl/site/artikel/ Help-ons-kunstmatige-intelligentie-testen/ 9163

But they are largely independent of each other when the data is high-dimensional

Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification

Conditional generative adversarial nets for convolutional face generation

Learning Feature Representations with K-Means

Removing Camera Shake from a Single Photograph

Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks

Log-likelihood, Parzen window estimates, and visual fidelity of generated samples

Evaluating Generative Models • How to measure model qualities?

• Training is done by backpropagation but difficult and usually unstable