Palette: Image-to-Image Diffusion Models (2021-11-10T00:00:00.000000Z)

TL;DR

A unified framework for image-to-image translation based on conditional diffusion models is developed and it is shown that a generalist, multi-task diffusion model performs as well or better than task-specific specialist counterparts.

Abstract

This paper develops a unified framework for image-to-image translation based on conditional diffusion models and evaluates this framework on four challenging image-to-image translation tasks, namely colorization, inpainting, uncropping, and JPEG restoration. Our simple implementation of image-to-image diffusion models outperforms strong GAN and regression baselines on all tasks, without task-specific hyper-parameter tuning, architecture customization, or any auxiliary loss or sophisticated new techniques needed. We uncover the impact of an L2 vs. L1 loss in the denoising diffusion objective on sample diversity, and demonstrate the importance of self-attention in the neural architecture through empirical studies. Importantly, we advocate a unified evaluation protocol based on ImageNet, with human evaluation and sample quality scores (FID, Inception Score, Classification Accuracy of a pre-trained ResNet-50, and Perceptual Distance against original images). We expect this standardized evaluation protocol to play a role in advancing image-to-image translation research. Finally, we show that a generalist, multi-task diffusion model performs as well or better than task-specific specialist counterparts. Check out https://diffusion-palette.github.io/ for an overview of the results and code.

Authors

David J. Fleet

7 papers

Chitwan Saharia

1 papers

William Chan

1 papers

TL;DR

Abstract

Authors

References115 items

Generative Adversarial Networks

Structured Denoising Diffusion Models in Discrete State-Spaces

Deep Edge-Aware Interactive Colorization against Color-Bleeding Effects

Variational Diffusion Models

WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation

Score-based Generative Modeling in Latent Space

Learning to Efficiently Sample from Diffusion Probabilistic Models

OCONet: Image Extrapolation by Object Completion

Cascaded Diffusion Models for High Fidelity Image Generation

Gotta Go Fast When Generating Data with Score-Based Models

Diffusion Models Beat GANs on Image Synthesis

Image Super-Resolution via Iterative Refinement

UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models

InOut: Diverse Image Outpainting via GAN Inversion

Scaling Local Self-Attention for Parameter Efficient Visual Backbones

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Improved Denoising Diffusion Probabilistic Models

Colorization Transformer

Zoom-to-Inpaint: Image Inpainting with High-Frequency Details

Score-Based Generative Modeling through Stochastic Differential Equations

DiffWave: A Versatile Diffusion Model for Audio Synthesis

WaveGrad: Estimating Gradients for Waveform Generation

Learning Gradient Fields for Shape Generation

Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser

Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations

NVAE: A Deep Hierarchical Variational Autoencoder

Denoising Diffusion Probabilistic Models

Improved Techniques for Training Score-Based Generative Models

UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation

Instance-Aware Image Colorization

Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting

Quantization Guided JPEG Artifact Correction

PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models

Very Long Natural Scenery Image Prediction by Outpainting

EdgeConnect: Structure Guided Image Inpainting using Edge Prediction

Boundless: Generative Adversarial Networks for Image Extension

Guided Image Generation with Conditional Invertible Neural Networks

Wide-Context Semantic Image Extrapolation

Classification Accuracy Score for Conditional Generative Models

Trinity of Pixel Enhancement: a Joint Solution for Demosaicking, Denoising and Super-Resolution

Few-Shot Unsupervised Image-to-Image Translation

Pluralistic Image Completion

FiNet: Compatible and Diverse Fashion Image Inpainting

Diversity-Sensitive Conditional Generative Adversarial Networks

Deep Universal Generative Adversarial Compression Artifact Removal

Glow: Generative Flow with Invertible 1x1 Convolutions

Free-Form Image Inpainting With Gated Convolution

Places: A 10 Million Image Database for Scene Recognition

Multi-level Wavelet-CNN for Image Restoration

Image Inpainting for Irregular Holes Using Partial Convolutions

Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning

Image Transformer

Generative Image Inpainting with Contextual Attention

StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation

NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study

Globally and locally consistent image completion

Rethinking Atrous Convolution for Semantic Image Segmentation

Attention is All you Need

PixColor: Pixel Recursive Colorization

Probabilistic Image Colorization

Deep Generative Adversarial Compression Artifact Removal

Improved Training of Wasserstein GANs

Pixel Recursive Super Resolution

Wasserstein GAN

PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

Learning Diverse Image Colorization

Image-to-Image Translation with Conditional Adversarial Networks

Unsupervised Cross-Domain Image Generation

Unrolled Generative Adversarial Networks

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising

Conditional Image Generation with PixelCNN Decoders

Density estimation using Real NVP

Colorful Image Colorization

Learning Representations for Automatic Colorization