Pretraining is All You Need for Image-to-Image Translation (2022-05-25T00:00:00.000000Z)

TL;DR

Examination of the proposed pretraining-based image-to-image translation (PITI) is shown to be capable of synthesizing images of unprecedented realism and faithfulness, and an adversarial training to enhance the texture synthesis in the diffusion model training is proposed.

Abstract

We propose to use pretraining to boost general image-to-image translation. Prior image-to-image translation methods usually need dedicated architectural design and train individual translation models from scratch, struggling for high-quality generation of complex scenes, especially when paired training data are not abundant. In this paper, we regard each image-to-image translation problem as a downstream task and introduce a simple and generic framework that adapts a pretrained diffusion model to accommodate various kinds of image-to-image translation. We also propose adversarial training to enhance the texture synthesis in the diffusion model training, in conjunction with normalized guidance sampling to improve the generation quality. We present extensive empirical comparison across various tasks on challenging benchmarks such as ADE20K, COCO-Stuff, and DIODE, showing the proposed pretraining-based image-to-image translation (PITI) is capable of synthesizing images of unprecedented realism and faithfulness.

Authors

Ting Zhang

4 papers

Dong Chen

8 papers

Hao Ouyang

3 papers

TL;DR

Abstract

Authors

References62 items

Classifier-Free Diffusion Guidance

Hierarchical Text-Conditional Image Generation with CLIP Latents

The Role of ImageNet Classes in Fréchet Inception Distance

data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

StyleSwin: Transformer-based GAN for High-resolution Image Generation

Masked Feature Prediction for Self-Supervised Visual Pre-Training

Multimodal Conditional Image Synthesis with Product-of-Experts GANs

Vector Quantized Diffusion Model for Text-to-Image Synthesis

Masked Autoencoders Are Scalable Vision Learners

Palette: Image-to-Image Diffusion Models

High-Fidelity GAN Inversion for Image Attribute Editing

BEiT: BERT Pre-Training of Image Transformers

TediGAN: Text-Guided Diverse Face Image Generation and Manipulation

CoCosNet v2: Full-Resolution Correspondence Learning for Image Translation

Cascaded Diffusion Models for High Fidelity Image Generation

M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers

GAN Prior Embedded Network for Blind Face Restoration in the Wild

Diffusion Models Beat GANs on Image Synthesis

Image Super-Resolution via Iterative Refinement

StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery

Designing a Practical Degradation Model for Deep Blind Image Super-Resolution

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Text-to-Image Generation

Image-to-Image Translation: Methods and Applications

Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Experts

Towards Real-World Blind Face Restoration with Generative Facial Prior

Taming Transformers for High-Resolution Image Synthesis

You Only Need Adversarial Supervision for Semantic Image Synthesis

Pre-Trained Image Processing Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Denoising Diffusion Implicit Models

Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications

Denoising Diffusion Probabilistic Models

Improved Techniques for Training Score-Based Generative Models

Language Models are Few-Shot Learners

Editing in Style: Uncovering the Local Semantics of GANs

Cross-Domain Correspondence Learning for Exemplar-Based Image Translation

Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation

PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models

A Simple Framework for Contrastive Learning of Visual Representations

Momentum Contrast for Unsupervised Visual Representation Learning

DIODE: A Dense Indoor and Outdoor DEpth Dataset

Trinity of Pixel Enhancement: a Joint Solution for Demosaicking, Denoising and Super-Resolution

Semantic Image Synthesis With Spatially-Adaptive Normalization

A Style-Based Generator Architecture for Generative Adversarial Networks

Large Scale GAN Training for High Fidelity Natural Image Synthesis

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation

Decoupled Weight Decay Regularization

Toward Multimodal Image-to-Image Translation

Scene Parsing through ADE20K Dataset

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

COCO-Stuff: Thing and Stuff Classes in Context

Image-to-Image Translation with Conditional Adversarial Networks

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Holistically-Nested Edge Detection

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Image smoothing via L0 gradient minimization

SDEdit: Image Synthesis and Editing with Stochastic Differential Equations

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

GENERATIVE ADVERSARIAL NETS

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names