PetsGAN: Rethinking Priors for Single Image Generation (2022-03-03T00:00:00.000000Z)

TL;DR

This paper digs into the single image generation problem and improves SinGAN by fully-utilization of internal and external priors, and designs a novel Prior-based end-to-end training GAN (PetsGAN), which is infused with internal prior and external prior to overcome the problems of SinGAN.

Abstract

Single image generation (SIG), described as generating diverse samples that have the same visual content as the given natural image, is first introduced by SinGAN, which builds a pyramid of GANs to progressively learn the internal patch distribution of the single image. It shows excellent performance in a wide range of image manipulation tasks. However, SinGAN has some limitations. Firstly, due to lack of semantic information, SinGAN cannot handle the object images well as it does on the scene and texture images. Secondly, the independent progressive training scheme is time-consuming and easy to cause artifacts accumulation. To tackle these problems, in this paper, we dig into the single image generation problem and improve SinGAN by fully-utilization of internal and external priors. The main contributions of this paper include: 1) We interpret single image generation from the perspective of the general generative task, that is, to learn a diverse distribution from the Dirac distribution composed of a single image. In order to solve this non-trivial problem, we construct a regularized latent variable model to formulate SIG. To the best of our knowledge, it is the first time to give a clear formulation and optimization goal of SIG, and all the existing methods for SIG can be regarded as special cases of this model. 2) We design a novel Prior-based end-to-end training GAN (PetsGAN), which is infused with internal prior and external prior to overcome the problems of SinGAN. For one thing, we employ the pre-trained GAN model to inject external prior for image generation, which can alleviate the problem of lack of semantic information and generate natural, reasonable and diverse samples, even for the object image. For another, we fully-utilize the internal prior by a differential Patch Matching module and an effective reconstruction network to generate consistent and realistic texture. 3) We construct abundant of qualitative and quantitative experiments on three datasets. The experimental results show our method surpasses other methods on both generated image quality, diversity, and training speed. Moreover, we apply our method to other image manipulation tasks (e.g., style transfer, harmonization) and the results further prove the effectiveness and efficiency of our method.

References49 items

Generative Adversarial Networks

SinIR: Efficient General Image Manipulation with Single Image Reconstruction

Patchwise Generative ConvNet: Training Energy-Based Models from a Single Natural Image for Internal Learning

One-Shot GAN: Learning to Generate Samples from Single Images and Videos

ExSinGAN: Learning an Explainable Generative Model from a Single Image

IMAGINE: Image Synthesis by Image-Guided Model Inversion

Drop the GAN: In Defense of Patches Nearest Neighbors as Single Image Generative Models

MOGAN: Morphologic-Structure-Aware Generative Learning From a Single Image

Towards Real-World Blind Face Restoration with Generative Facial Prior

Positional Encoding as Spatial Inductive Bias in GANs

Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample

Transforming and Projecting Images into Class-conditional Generative Networks

Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation

Improved Techniques for Training Single-Image GANs

Image Processing Using Multi-Code GAN Prior

Analyzing and Improving the Image Quality of StyleGAN

InGAN: Capturing and Retargeting the “DNA” of a Natural Image

SinGAN: Learning a Generative Model From a Single Natural Image

Image Super-Resolution by Neural Texture Transfer

Diversity-Sensitive Conditional Generative Adversarial Networks

A Style-Based Generator Architecture for Generative Adversarial Networks

“Double-DIP”: Unsupervised Image Decomposition via Coupled Deep-Image-Priors

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Non-stationary texture synthesis by adversarial expansion

Which Training Methods for GANs do actually Converge?

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Zero-Shot Super-Resolution Using Deep Internal Learning

Deep Image Prior

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Wasserstein Generative Adversarial Networks

Improved Training of Wasserstein GANs

Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis

Texture Synthesis with Spatial Generative Adversarial Networks

Image-to-Image Translation with Conditional Adversarial Networks

Generative Visual Manipulation on the Natural Image Manifold

Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Pixel Recurrent Neural Networks

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop