MaskGAN: Towards Diverse and Interactive Facial Image Manipulation (2019-07-27T00:00:00.000000Z)

TL;DR

This work proposes a novel framework termed MaskGAN, enabling diverse and interactive face manipulation, and finds that semantic masks serve as a suitable intermediate representation for flexible face manipulation with fidelity preservation.

Abstract

Facial image manipulation has achieved great progress in recent years. However, previous methods either operate on a predefined set of face attributes or leave users little freedom to interactively manipulate images. To overcome these drawbacks, we propose a novel framework termed MaskGAN, enabling diverse and interactive face manipulation. Our key insight is that semantic masks serve as a suitable intermediate representation for flexible face manipulation with fidelity preservation. MaskGAN has two main components: 1) Dense Mapping Network (DMN) and 2) Editing Behavior Simulated Training (EBST). Specifically, DMN learns style mapping between a free-form user modified mask and a target image, enabling diverse generation results. EBST models the user editing behavior on the source mask, making the overall framework more robust to various manipulated inputs. Specifically, it introduces dual-editing consistency as the auxiliary supervision signal. To facilitate extensive studies, we construct a large-scale high-resolution face dataset with fine-grained mask annotations named CelebAMask-HQ. MaskGAN is comprehensively evaluated on two challenging tasks: attribute transfer and style copy, demonstrating superior performance over other state-of-the-art methods. The code, models, and dataset are available at https://github.com/switchablenorms/CelebAMask-HQ.

Authors

Ziwei Liu

27 papers

Ping Luo

22 papers

Cheng-Han Lee

1 papers

TL;DR

Abstract

Authors

References59 items

Towards Photo-Realistic Virtual Try-On by Adaptively Generating↔Preserving Image Content

One-shot Face Reenactment

Mask-Guided Portrait Editing With Conditional GANs

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Semantic Image Synthesis With Spatially-Adaptive Normalization

SC-FEGAN: Face Editing Generative Adversarial Network With User’s Sketch and Color

paGAN

Warp-guided GANs for single-photo facial animation

PIVTONS: Pose Invariant Virtual Try-On Shoe with Conditional Image Completion

Instance-level Facial Attributes Transfer with Geometry-Aware Flow

Toward Characteristic-Preserving Image-based Virtual Try-On Network

Free-Form Image Inpainting With Gated Convolution

Attribute Augmented Convolutional Neural Network for Face Hallucination

Image Inpainting for Irregular Holes Using Partial Convolutions

Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform

ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributes

Generative Image Inpainting with Contextual Attention

ArcFace: Additive Angular Margin Loss for Deep Face Recognition

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation

VITON: An Image-Based Virtual Try-on Network

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Fine-Grained Head Pose Estimation Without Keypoints

Photographic Image Synthesis with Cascaded Refinement Networks

Fader Networks: Manipulating Images by Sliding Attributes

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Learning to Discover Cross-Domain Relations with Generative Adversarial Networks

Unsupervised Image-to-Image Translation Networks

Semantic Facial Expression Editing using Autoencoded Flow

Image-to-Image Translation with Conditional Adversarial Networks

Invertible Conditional GANs for image editing

Deep Identity-aware Transfer of Facial Attributes

Instance Normalization: The Missing Ingredient for Fast Stylization

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Deep Residual Learning for Image Recognition

Semantic Image Segmentation via Deep Parsing Network

U-Net: Convolutional Networks for Biomedical Image Segmentation

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

Deep Learning Face Attributes in the Wild

Conditional Generative Adversarial Nets

Very Deep Convolutional Networks for Large-Scale Image Recognition

FaceWarehouse: A 3D Facial Expression Database for Visual Computing

Auto-Encoding Variational Bayes

Interactive Facial Feature Localization

ImageNet: A large-scale hierarchical image database

Compositing digital images

Stargan: Unified generative Metric Attribute cls. acc(%) Seg(%) FID MaskGAN-concat

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

GENERATIVE ADVERSARIAL NETS

Linear (1024)ˋL inear (32768)

Our design of discriminator also follows Pix2PixHD [27] which utilize PatchGAN [8]. We concatenate the masks and images as inputs to realize conditional GAN

The detailed architecture of SFT Layer is shown in Fig. 10 which does not use any normalization for all layers

The architecture of MaskVAE is similar to UNet [23] without skip-connection. Detailed architectures of Enc VAE and Dec VAE are shown in Fig. 8 which uses BN for all layers

Alpha Blender also follows the desing of Pix2PixHD but only downsampling three times and using three residual blocks. The detailed architecture is as following: c7s1−32

Spatial-Aware Style Encoder consists of two branches for receiving both style and spatial information. Two fuse two different domains

Additional Visual Results In Fig. 11, we show additional visual results of attribute transfer for specific attribute: Smiling. We compare our MaskGAN with state-of-the art methods including

We choose the architecture of Pix2PixHD [27] as Image Generation Backbone. The detailed architecture is as following: c7s1−64

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names