Generative Modeling by Estimating Gradients of the Data Distribution

Published in

Neural Information Processing Systems(2019)

External Links:

Generate Graph

TL;DR

A new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching, which allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons.

Abstract

We introduce a new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching. Because gradients can be ill-defined and hard to estimate when the data resides on low-dimensional manifolds, we perturb the data with different levels of Gaussian noise, and jointly estimate the corresponding scores, i.e., the vector fields of gradients of the perturbed data distribution for all noise levels. For sampling, we propose an annealed Langevin dynamics where we use gradients corresponding to gradually decreasing noise levels as the sampling process gets closer to the data manifold. Our framework allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Our models produce samples comparable to GANs on MNIST, CelebA and CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.87 on CIFAR-10. Additionally, we demonstrate that our models learn effective representations via image inpainting experiments.

Authors

Stefano Ermon

21 papers

Yang Song

2 papers

References68 items

Sliced Score Matching: A Scalable Approach to Density and Score Estimation

On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models

A Style-Based Generator Architecture for Generative Adversarial Networks

Convolutional adaptive denoising autoencoders for hierarchical feature extraction

Learning deep kernels for exponential family densities

Generative Modeling by Estimating Gradients of the Data Distribution

Published in

Neural Information Processing Systems(2019)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Stefano Ermon

21 papers

Yang Song

2 papers

References68 items

Sliced Score Matching: A Scalable Approach to Density and Score Estimation

On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models

A Style-Based Generator Architecture for Generative Adversarial Networks

Convolutional adaptive denoising autoencoders for hierarchical feature extraction

Learning deep kernels for exponential family densities

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Learning Implicit Generative Models with the Method of Learned Moments

Autoregressive Quantile Networks for Generative Modeling

Deep Energy Estimator Networks

Spectral Normalization for Generative Adversarial Networks

Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net

PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Wasserstein Generative Adversarial Networks

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

A-NICE-MC: Adversarial Training for MCMC

Dilated Residual Networks

Good Semi-supervised Learning That Requires a Bad GAN

Improved Training of Wasserstein GANs

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Learning to Generate Samples from Noise through Infusion Training

Count-Based Exploration with Neural Density Models

RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation

A Learned Representation For Artistic Style

WaveNet: A Generative Model for Raw Audio

Instance Normalization: The Missing Ingredient for Fast Stylization

Conditional Image Generation with PixelCNN Decoders

Improved Techniques for Training GANs

Generative Adversarial Imitation Learning

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization

A Kernelized Stein Discrepancy for Goodness-of-fit Tests

Pixel Recurrent Neural Networks

Multi-Scale Context Aggregation by Dilated Convolutions

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

U-Net: Convolutional Networks for Biomedical Image Segmentation

GSNs : Generative Stochastic Networks

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Deep Learning Face Attributes in the Wild

Adaptive Noise Schedule for Denoising Autoencoder

NICE: Non-linear Independent Components Estimation

Semi-supervised Learning with Deep Generative Models

Scheduled denoising autoencoders

Stochastic Gradient Hamiltonian Monte Carlo

Auto-Encoding Variational Bayes

Generating Sequences With Recurrent Neural Networks

Generalized Denoising Auto-Encoders as Generative Models

A Connection Between Score Matching and Denoising Autoencoders

Bayesian Learning via Stochastic Gradient Langevin Dynamics

MCMC Using Hamiltonian Dynamics

Least Squares Estimation Without Priors or Supervision

Regularized estimation of image statistics by Score Matching

Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

Minimum Probability Flow Learning

On integral probability metrics, φ-divergences and binary classification

Learning to be Bayesian without Supervision

Estimation of Non-Normalized Statistical Models by Score Matching

Laplacian Eigenmaps for Dimensionality Reduction and Data Representation

Training Products of Experts by Minimizing Contrastive Divergence

Nonlinear dimensionality reduction by locally linear embedding.

Annealed importance sampling

Low-dimensional procedure for the characterization of human faces.

Optimization by Simulated Annealing

Estimation of the Mean of a Multivariate Normal Distribution

Implicit Generation and Generalization with Energy Based Models

GENERATIVE ADVERSARIAL NETS

Learning Multiple Layers of Features from Tiny Images

An empirical bayes estimator of the mean of a normal population

Field of Study

Computer ScienceMathematics

Venue Information

Name

Neural Information Processing Systems

Type

conference

URL

http://neurips.cc/

Alternate Names

Neural Inf Process Syst
NeurIPS
NIPS