Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation (2020-03-30T00:00:00.000000Z)

TL;DR

This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images and allows the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN.

Abstract

Learning a good image prior is a long-term goal for image restoration and manipulation. While existing methods like deep image prior (DIP) capture low-level image statistics, there are still gaps toward an image prior that captures rich image semantics including color, spatial coherence, textures, and high-level concepts. This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. As shown in Fig. 1, the deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images. It also enables diverse image manipulation including random jittering, image morphing, and category transfer. Such highly flexible restoration and manipulation are made possible through relaxing the assumption of existing GAN inversion methods, which tend to fix the generator. Notably, we allow the generator to be fine-tuned on-the-fly in a progressive manner regularized by feature distance obtained by the discriminator in GAN. We show that these easy-to-implement and practical changes help preserve the reconstruction to remain in the manifold of nature images, and thus lead to more precise and faithful reconstruction for real images. Code is available at https://github.com/XingangPan/deep-generative-prior.

Authors

Chen Change Loy

72 papers

Dahua Lin

31 papers

P. Luo

15 papers

TL;DR

Abstract

Authors

References63 items

Generative Adversarial Networks

Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

Closed-Form Factorization of Latent Semantics in GANs

Transforming and Projecting Images into Class-conditional Generative Networks

Inclusive GAN: Improving Data and Minority Coverage in Generative Models

GANSpace: Discovering Interpretable GAN Controls

In-Domain GAN Inversion for Real Image Editing

PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models

Real or Not Real, that is the Question

Image Processing Using Multi-Code GAN Prior

Image2StyleGAN++: How to Edit the Embedded Images?

Seeing What a GAN Cannot Generate

Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis

Interpreting the Latent Space of GANs for Semantic Face Editing

Semantic photo manipulation with a generative image prior

Image-Adaptive GAN based Reconstruction

SinGAN: Learning a Generative Model From a Single Natural Image

Source Generator Attribution via Inversion

Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?

Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis

A Style-Based Generator Architecture for Generative Adversarial Networks

Deep Network Interpolation for Continuous Imagery Effect Transition

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Latent Convolutional Models

Places: A 10 Million Image Database for Scene Recognition

Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models

Generative Image Inpainting with Contextual Attention

Zero-Shot Super-Resolution Using Deep Internal Learning

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Deep Image Prior

StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation

Automatic differentiation in PyTorch

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Deep Mean-Shift Priors for Image Restoration

Learning Deep CNN Denoiser Prior for Image Restoration

Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks

Adversarial Transformation Networks: Learning to Generate Adversarial Examples

Towards Diverse and Natural Image Descriptions via a Conditional GAN

Inverting the Generator of a Generative Adversarial Network

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Generative Visual Manipulation on the Natural Image Manifold

Semantic Image Inpainting with Deep Generative Models

Improved Techniques for Training GANs

Adversarial Feature Learning

Colorful Image Colorization

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Learning Representations for Automatic Colorization

Deep Residual Learning for Image Recognition

Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Image Super-Resolution Using Deep Convolutional Networks

Adam: A Method for Stochastic Optimization

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Making a “Completely Blind” Image Quality Analyzer

ImageNet: A large-scale hierarchical image database

Single image haze removal using dark channel prior

Fields of Experts: a framework for learning image priors

Prior Learning and Gibbs Reaction-Diffusion

Nonlinear total variation based noise removal algorithms

Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images

Exploiting deep generative prior for versatile image restoration and manipulation

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Category transfer. The red box shows the target, and the blue box shows the reconstruction. Others are category transfer results

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names