Multimodal Unsupervised Image-to-Image Translation (2018-04-12T00:00:00.000000Z)

TL;DR

A Multimodal Unsupervised Image-to-image Translation (MUNIT) framework that assumes that the image representation can be decomposed into a content code that is domain-invariant, and a style code that captures domain-specific properties.

Abstract

Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any pairs of corresponding images. While this conditional distribution is inherently multimodal, existing approaches make an overly simplified assumption, modeling it as a deterministic one-to-one mapping. As a result, they fail to generate diverse outputs from a given source domain image. To address this limitation, we propose a Multimodal Unsupervised Image-to-image Translation (MUNIT) framework. We assume that the image representation can be decomposed into a content code that is domain-invariant, and a style code that captures domain-specific properties. To translate an image to another domain, we recombine its content code with a random style code sampled from the style space of the target domain. We analyze the proposed framework and establish several theoretical results. Extensive experiments with comparisons to the state-of-the-art approaches further demonstrates the advantage of the proposed framework. Moreover, our framework allows users to control the style of translation outputs by providing an example style image. Code and pretrained models are available at this https URL

Authors

Serge J. Belongie

28 papers

Xun Huang

4 papers

Ming-Yu Liu

13 papers

TL;DR

Abstract

Authors

References87 items

Diverse Image-to-Image Translation via Disentangled Representations

Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data

A Closed-form Solution to Photorealistic Image Stylization

Identifying Analogies Across Domains

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

ComboGAN: Unrestrained Scalability for Image Domain Translation

Unsupervised Multi-Domain Image Translation with Domain-Specific Encoders/Decoders

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation

XGAN: Unsupervised Image-to-Image Translation for many-to-many Mappings

Wasserstein Auto-Encoders

Toward Multimodal Image-to-Image Translation

Automatic differentiation in PyTorch

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Triangle Generative Adversarial Networks

ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching

The Role of Minimal Complexity Functions in Unsupervised Learning of Semantic Mappings

PixelNN: Example-based Image Synthesis

Generative Semantic Manipulation with Contrasting GAN

Photographic Image Synthesis with Cascaded Refinement Networks

MoCoGAN: Decomposing Motion and Content for Video Generation

Wasserstein Generative Adversarial Networks

Decomposing Motion and Content for Natural Video Sequence Prediction

Variational Approaches for Auto-Encoding Generative Adversarial Networks

One-Sided Unsupervised Domain Mapping

Unsupervised Learning of Disentangled Representations from Video

Universal Style Transfer via Feature Transforms

Semantically Decomposing the Latent Spaces of Generative Adversarial Networks

VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning

Exploring the structure of a real-time, arbitrary neural artistic stylization network

Style Transfer from Non-Parallel Text by Cross-Alignment

Unsupervised Creation of Parameterized Avatars

Multi-agent Diverse Generative Adversarial Networks

DualGAN: Unsupervised Dual Learning for Image-to-Image Translation

BEGAN: Boundary Equilibrium Generative Adversarial Networks

ZM-Net: Real-time Zero-shot Image Manipulation Network

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Learning to Discover Cross-Domain Relations with Generative Adversarial Networks

Unsupervised Image-to-Image Translation Networks

Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis

Learning from Simulated and Unsupervised Images through Adversarial Training

Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

Stacked Generative Adversarial Networks

Fast Patch-based Style Transfer of Arbitrary Style

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

Image-to-Image Translation with Conditional Adversarial Networks

Least Squares Generative Adversarial Networks

Disentangling factors of variation in deep representation using adversarial training

Unsupervised Cross-Domain Image Generation

LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation

beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

A Learned Representation For Artistic Style

Generative Visual Manipulation on the Natural Image Manifold

Energy-based Generative Adversarial Network

Domain Separation Networks

Image Style Transfer Using Convolutional Neural Networks

The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes

Coupled Generative Adversarial Networks

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Improved Techniques for Training GANs

Adversarially Learned Inference

Adversarial Feature Learning

Context Encoders: Feature Learning by Inpainting

The Cityscapes Dataset for Semantic Urban Scene Understanding

Colorful Image Colorization

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Generative Image Modeling Using Style and Structure Adversarial Networks

Revisiting Batch Normalization For Practical Domain Adaptation

Generating Images with Perceptual Similarity Metrics based on Deep Networks

Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis

Autoencoding beyond pixels using a learned similarity metric

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Adversarial Autoencoders

Deep multi-scale video prediction beyond mean square error

Double bubbles sans toil and trouble