Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders

Published in

arXiv.org(2016)

External Links:

Generate Graph

TL;DR

It is shown that a heuristic called minimum information constraint that has been shown to mitigate this effect in VAEs can also be applied to improve unsupervised clustering performance with this variant of the variational autoencoder model with a Gaussian mixture as a prior distribution.

Abstract

We study a variant of the variational autoencoder model (VAE) with a Gaussian mixture as a prior distribution, with the goal of performing unsupervised clustering through deep generative models. We observe that the known problem of over-regularisation that has been shown to arise in regular VAEs also manifests itself in our model and leads to cluster degeneracy. We show that a heuristic called minimum information constraint that has been shown to mitigate this effect in VAEs can also be applied to improve unsupervised clustering performance with our model. Furthermore we analyse the effect of this heuristic and provide an intuition of the various processes with the help of visualizations. Finally, we demonstrate the performance of our model on synthetic data, MNIST and SVHN, showing that the obtained clusters are distinct, interpretable and result in achieving competitive performance on unsupervised clustering to the state-of-the-art results.

Authors

Kai Arulkumaran

4 papers

M. Garnelo

4 papers

P. Mediano

2 papers

References35 items

Variational Lossy Autoencoder

Stochastic Backpropagation through Mixture Density Distributions

Tagger: Deep Unsupervised Perceptual Grouping

Early Visual Concept Learning with Unsupervised Deep Learning

Improved Variational Inference with Inverse Autoregressive Flow

Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders

Published in

arXiv.org(2016)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Kai Arulkumaran

4 papers

M. Garnelo

4 papers

P. Mediano

2 papers

References35 items

Variational Lossy Autoencoder

Stochastic Backpropagation through Mixture Density Distributions

Tagger: Deep Unsupervised Perceptual Grouping

Early Visual Concept Learning with Unsupervised Deep Learning

Improved Variational Inference with Inverse Autoregressive Flow

Nat Dilokthanakul

2 papers

Hugh Salimbeni

2 papers

M. Shanahan

2 papers

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

Composing graphical models with neural networks for structured representations and fast inference

Ladder Variational Autoencoders

How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

Local Expectation Gradients for Black Box Variational Inference

Unsupervised Deep Embedding for Clustering Analysis

Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks

Generating Sentences from a Continuous Space

Adversarial Autoencoders

A Recurrent Latent Variable Model for Sequential Data

DRAW: A Recurrent Neural Network For Image Generation

Adam: A Method for Stochastic Optimization

Semi-supervised Learning with Deep Generative Models

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

Demystifying Information-Theoretic Clustering

Graphical Models, Exponential Families, and Variational Inference

Pattern Recognition and Machine Learning

Elements of Information Theory

Likelihood ratio gradient estimation for stochastic systems

GENERATIVE ADVERSARIAL NETS

Elbo surgery: yet another way to carve up the variational evidence lower bound

Stochastic Video Prediction with Conditional Density Estimation

Under Review as a Conference Paper at Iclr 2017 Delving into Transferable Adversarial Ex- Amples and Black-box Attacks

Data Clustering : Algorithms and Applications

Reading Digits in Natural Images with Unsupervised Feature Learning

Gradient-based learning applied to document recognition

we use Adam (Kingma & Ba, 2014) with a learning rate of 10 −4 and standard hyperparameter values β 1 = 0.9, β 2 = 0.999 and = 10 −8 . The model architectures used in our experiments are shown

Field of Study

MathematicsComputer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

arXiv.org

Type

URL

https://arxiv.org

Alternate Names

ArXiv

TL;DR

Abstract

Authors

References35 items

Variational Lossy Autoencoder

Stochastic Backpropagation through Mixture Density Distributions

Tagger: Deep Unsupervised Perceptual Grouping

Early Visual Concept Learning with Unsupervised Deep Learning

Improved Variational Inference with Inverse Autoregressive Flow

TL;DR

Abstract

Authors

References35 items

Variational Lossy Autoencoder

Stochastic Backpropagation through Mixture Density Distributions

Tagger: Deep Unsupervised Perceptual Grouping

Early Visual Concept Learning with Unsupervised Deep Learning

Improved Variational Inference with Inverse Autoregressive Flow

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

Composing graphical models with neural networks for structured representations and fast inference

Ladder Variational Autoencoders

How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

Local Expectation Gradients for Black Box Variational Inference

Unsupervised Deep Embedding for Clustering Analysis

Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks

Generating Sentences from a Continuous Space

Adversarial Autoencoders

A Recurrent Latent Variable Model for Sequential Data

DRAW: A Recurrent Neural Network For Image Generation

Adam: A Method for Stochastic Optimization

Semi-supervised Learning with Deep Generative Models

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

Demystifying Information-Theoretic Clustering

Graphical Models, Exponential Families, and Variational Inference

Pattern Recognition and Machine Learning

Elements of Information Theory

Likelihood ratio gradient estimation for stochastic systems

GENERATIVE ADVERSARIAL NETS

Elbo surgery: yet another way to carve up the variational evidence lower bound

Stochastic Video Prediction with Conditional Density Estimation

Under Review as a Conference Paper at Iclr 2017 Delving into Transferable Adversarial Ex- Amples and Black-box Attacks

Data Clustering : Algorithms and Applications

Reading Digits in Natural Images with Unsupervised Feature Learning

Cover and

Gradient-based learning applied to document recognition

we use Adam (Kingma & Ba, 2014) with a learning rate of 10 −4 and standard hyperparameter values β 1 = 0.9, β 2 = 0.999 and = 10 −8 . The model architectures used in our experiments are shown

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names