Score-Based Generative Modeling through Stochastic Differential Equations

Published in

International Conference on Learning Representa...(2020)

External Links:

Generate Graph

TL;DR

This work presents a stochastic differential equation (SDE) that smoothly transforms a complex data distribution to a known prior distribution by slowly injecting noise, and a corresponding reverse-time SDE that transforms the prior distribution back into the data distribution by Slowly removing the noise.

Abstract

Creating noise from data is easy; creating data from noise is generative modeling. We present a stochastic differential equation (SDE) that smoothly transforms a complex data distribution to a known prior distribution by slowly injecting noise, and a corresponding reverse-time SDE that transforms the prior distribution back into the data distribution by slowly removing the noise. Crucially, the reverse-time SDE depends only on the time-dependent gradient field (\aka, score) of the perturbed data distribution. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks, and use numerical SDE solvers to generate samples. We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic modeling, allowing for new sampling procedures and new modeling capabilities. In particular, we introduce a predictor-corrector framework to correct errors in the evolution of the discretized reverse-time SDE. We also derive an equivalent neural ODE that samples from the same distribution as the SDE, but additionally enables exact likelihood computation, and improved sampling efficiency. In addition, we provide a new way to solve inverse problems with score-based models, as demonstrated with experiments on class-conditional generation, image inpainting, and colorization. Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9.89 and FID of 2.20, a competitive likelihood of 2.99 bits/dim, and demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.

Authors

Score-Based Generative Modeling through Stochastic Differential Equations

Published in

International Conference on Learning Representa...(2020)

External Links:

Generate Graph

TL;DR

Abstract

Authors

References65 items

DiffWave: A Versatile Diffusion Model for Audio Synthesis

Adversarial score matching and improved sampling for image generation

WaveGrad: Estimating Gradients for Waveform Generation

Learning Gradient Fields for Shape Generation

Efficient Learning of Generative Models via Finite-Difference Score Matching

On Linear Identifiability of Learned Representations

Denoising Diffusion Probabilistic Models

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Improved Techniques for Training Score-Based Generative Models

Training Generative Adversarial Networks with Limited Data

Interacting Particle Solutions of Fokker–Planck Equations Through Gradient–Log–Density Estimation

Permutation Invariant Graph Generation via Score-Based Generative Modeling

Analyzing and Improving the Image Quality of StyleGAN

MintNet: Building Invertible Neural Networks with Masked Convolutions

Generative Modeling by Estimating Gradients of the Data Distribution

Generating Diverse High-Fidelity Images with VQ-VAE-2

Residual Flows for Invertible Generative Modeling

Sliced Score Matching: A Scalable Approach to Density and Score Estimation

Making Convolutional Networks Shift-Invariant Again

Applied Stochastic Differential Equations

Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design

A Style-Based Generator Architecture for Generative Adversarial Networks

Invertible Residual Networks

(Preprint)

FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Glow: Generative Flow with Invertible 1x1 Convolutions

Neural Ordinary Differential Equations

Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Learning to Generate Samples from Noise through Infusion Training

Density estimation using Real NVP

Wide Residual Networks

LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Deep Learning Face Attributes in the Wild

Tweedie’s Formula and Selection Bias

A Connection Between Score Matching and Denoising Autoencoders

MCMC Using Hamiltonian Dynamics

Estimation of Non-Normalized Statistical Models by Score Matching

REPRESENTATIONS OF KNOWLEDGE IN COMPLEX SYSTEMS

Numerical Solution of Stochastic Differential Equations

Numerical continuation methods - an introduction

Reverse-time diffusion equation models

A family of embedded Runge-Kutta formulae

́1 is one representative term in the ELBO objective (see Eq

pdatapxq. In our experiments, we let β̄min “ 0.1 and β̄max “ 20 to match the settings

Upsampling and downsampling images with anti-aliasing based on Finite Impulse Response (FIR) (Zhang, 2019)

The basic NCSN++ model with 4 residual blocks achieves an FID of 2.45 on CIFAR-10. Here in order to match the convention used in Karras et al

Implicit Generation and Modeling with Energy Based Models

2018) and StyleGAN (Karras et al., 2019)

GENERATIVE ADVERSARIAL NETS

Rescaling all skip connections by 1 { ? 2. This has been demonstrated effective in several bestin-class GAN models, including ProgressiveGAN

Replacing the original residual blocks in DDPM with residual blocks from BigGAN

2018), a task that was previously only achievable by some GAN models and

Stochastic differential equations

Learning Multiple Layers of Features from Tiny Images

A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines

The Eigenvalues of Mega-dimensional Matrices

Correlation functions and computer simulations

CLASS-CONDITIONAL SAMPLING

Since the forward SDE is tractable, we can easily create a pair of training data pxptq, yq by first sampling pxp0q, yq from a dataset and then obtaining xptq

Increasing the number of residual blocks per resolution from 2 to 4

the FID from 2.38 to 2.2 by doubling the number of residual blocks per resolution, resulting in the model denoted as NCSN++ cont

All sampling methods we have discussed so far can be applied to the conditional reverse-time SDE for sample generation

TL;DR

Abstract

Authors

TL;DR

Abstract

Authors

References65 items

DiffWave: A Versatile Diffusion Model for Audio Synthesis

Adversarial score matching and improved sampling for image generation

WaveGrad: Estimating Gradients for Waveform Generation

Learning Gradient Fields for Shape Generation

Efficient Learning of Generative Models via Finite-Difference Score Matching

On Linear Identifiability of Learned Representations

Denoising Diffusion Probabilistic Models

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Improved Techniques for Training Score-Based Generative Models

Training Generative Adversarial Networks with Limited Data

Interacting Particle Solutions of Fokker–Planck Equations Through Gradient–Log–Density Estimation

Permutation Invariant Graph Generation via Score-Based Generative Modeling

Analyzing and Improving the Image Quality of StyleGAN

MintNet: Building Invertible Neural Networks with Masked Convolutions

Generative Modeling by Estimating Gradients of the Data Distribution

Generating Diverse High-Fidelity Images with VQ-VAE-2

Residual Flows for Invertible Generative Modeling

Sliced Score Matching: A Scalable Approach to Density and Score Estimation

Making Convolutional Networks Shift-Invariant Again

Applied Stochastic Differential Equations

Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design

A Style-Based Generator Architecture for Generative Adversarial Networks

Invertible Residual Networks

(Preprint)

FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Glow: Generative Flow with Invertible 1x1 Convolutions

Neural Ordinary Differential Equations

Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Learning to Generate Samples from Noise through Infusion Training

Density estimation using Real NVP

Wide Residual Networks

LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Deep Learning Face Attributes in the Wild

Tweedie’s Formula and Selection Bias

A Connection Between Score Matching and Denoising Autoencoders

MCMC Using Hamiltonian Dynamics

Estimation of Non-Normalized Statistical Models by Score Matching

REPRESENTATIONS OF KNOWLEDGE IN COMPLEX SYSTEMS

Numerical Solution of Stochastic Differential Equations

Numerical continuation methods - an introduction

Reverse-time diffusion equation models

A family of embedded Runge-Kutta formulae

́1 is one representative term in the ELBO objective (see Eq

pdatapxq. In our experiments, we let β̄min “ 0.1 and β̄max “ 20 to match the settings

Upsampling and downsampling images with anti-aliasing based on Finite Impulse Response (FIR) (Zhang, 2019)

The basic NCSN++ model with 4 residual blocks achieves an FID of 2.45 on CIFAR-10. Here in order to match the convention used in Karras et al

Implicit Generation and Modeling with Energy Based Models

2018) and StyleGAN (Karras et al., 2019)

GENERATIVE ADVERSARIAL NETS

Rescaling all skip connections by 1 { ? 2. This has been demonstrated effective in several bestin-class GAN models, including ProgressiveGAN

Replacing the original residual blocks in DDPM with residual blocks from BigGAN

2018), a task that was previously only achievable by some GAN models and

Stochastic differential equations

Learning Multiple Layers of Features from Tiny Images

A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines

The Eigenvalues of Mega-dimensional Matrices

Correlation functions and computer simulations

CLASS-CONDITIONAL SAMPLING

Since the forward SDE is tractable, we can easily create a pair of training data pxptq, yq by first sampling pxp0q, yq from a dataset and then obtaining xptq

Increasing the number of residual blocks per resolution from 2 to 4

the FID from 2.38 to 2.2 by doubling the number of residual blocks per resolution, resulting in the model denoted as NCSN++ cont

All sampling methods we have discussed so far can be applied to the conditional reverse-time SDE for sample generation

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL