Improved Training of Wasserstein GANs (2017-03-31T00:00:00.000000Z)

TL;DR

This work proposes an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input, which performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning.

Abstract

Generative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.

Authors

Aaron C. Courville

35 papers

Vincent Dumoulin

7 papers

Martín Arjovsky

3 papers

TL;DR

Abstract

Authors

References37 items

Approximation and Convergence Properties of Generative Adversarial Learning

BEGAN: Boundary Equilibrium Generative Adversarial Networks

Recurrent Topic-Transition GAN for Visual Paragraph Generation

Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets

Boundary-Seeking Generative Adversarial Networks

Maximum-Likelihood Augmented Discrete Generative Adversarial Networks

Calibrating Energy-based Generative Adversarial Networks

Wasserstein GAN

Adversarial Learning for Neural Dialogue Generation

Towards Principled Methods for Training Generative Adversarial Networks

Stacked Generative Adversarial Networks

Improved generator objectives for GANs

Least Squares Generative Adversarial Networks

Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning

Improving Generative Adversarial Networks with Denoising Feature Matching

Unrolled Generative Adversarial Networks

On the Quantitative Analysis of Decoder-Based Generative Models

Categorical Reparameterization with Gumbel-Softmax

The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables

Conditional Image Synthesis with Auxiliary Classifier GANs

SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient

Instance Normalization: The Missing Ingredient for Fast Stylization

Layer Normalization

Conditional Image Generation with PixelCNN Decoders

Improved Techniques for Training GANs

Adversarially Learned Inference

f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop

Adam: A Method for Stochastic Optimization

On distinguishability criteria for estimating generative models

One billion word benchmark for measuring progress in statistical language modeling

Optimal Transport: Old and New

GENERATIVE ADVERSARIAL NETS

Learning Multiple Layers of Features from Tiny Images

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Field of Study

Venue Information

Name

Type

URL

Alternate Names