Importance Weighted Autoencoders (2015-09-01T00:00:00.000000Z)

TL;DR

The importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting, shows empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log- likelihood on density estimation benchmarks.

Abstract

The variational autoencoder (VAE; Kingma, Welling (2014)) is a recently proposed generative model pairing a top-down generative network with a bottom-up recognition network which approximates posterior inference. It typically makes strong assumptions about posterior inference, for instance that the posterior distribution is approximately factorial, and that its parameters can be approximated with nonlinear regression from the observations. As we show empirically, the VAE objective can lead to overly simplified representations which fail to use the network's entire modeling capacity. We present the importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting. In the IWAE, the recognition network uses multiple samples to approximate the posterior, giving it increased flexibility to model complex posteriors which do not fit the VAE modeling assumptions. We show empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log-likelihood on density estimation benchmarks.

Authors

R. Salakhutdinov

32 papers

Yuri Burda

4 papers

R. Grosse

5 papers

TL;DR

Abstract

Authors

References35 items

Visual Attention

Scaling up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix

Variational Inference with Normalizing Flows

Training generative neural networks via Maximum Mean Discrepancy optimization

Deep Convolutional Inverse Graphics Network

DRAW: A Recurrent Neural Network For Image Generation

Generative Moment Matching Networks

Accurate and conservative estimates of MRF log-likelihood using reverse annealing

Adam: A Method for Stochastic Optimization

Semi-supervised Learning with Deep Generative Models

Reweighted Wake-Sleep

Neural Variational Inference and Learning in Belief Networks

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

Learning Stochastic Feedforward Neural Networks

One-shot learning by inverting a compositional causal process

Deep AutoRegressive Networks

Efficient Learning of Deep Boltzmann Machines

Understanding the difficulty of training deep feedforward neural networks

Deep Boltzmann Machines

Evaluating probabilities under high-dimensional latent variable models

On the quantitative analysis of deep belief networks

Studies in Lower Bounding Probabilities of Evidence using the Markov Inequality

A Fast Learning Algorithm for Deep Belief Nets

The Helmholtz Machine

Connectionist Learning of Belief Networks

Simple statistical gradient-following algorithms for connectionist reinforcement learning

Information processing in dynamical systems: foundations of harmony theory

GENERATIVE ADVERSARIAL NETS

Markov Chain Monte Carlo and Variational Inference: Bridging the Gap

The Neural Autoregressive Distribution Estimator

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Gradient-based learning applied to document recognition

We prove each in turn

Under review as a conference paper at ICLR 2019 softmax Transpose Updated Memory Memory embedded input linear concat Attention Weights concat

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names