Max-Margin Deep Generative Models for (Semi-)Supervised Learning (2016-11-22T00:00:00.000000Z)

TL;DR

Max-margin deep generative models (mmDGMs) and a class-conditional variant (mmDCGMs), which explore the strongly discriminative principle of max-margin learning to improve the predictive performance of DGMs in both supervised and semi-supervised learning, while retaining the generative capability.

Abstract

Deep generative models (DGMs) can effectively capture the underlying distributions of complex data by learning multilayered representations and performing inference. However, it is relatively insufficient to boost the discriminative ability of DGMs. This paper presents max-margin deep generative models (mmDGMs) and a class-conditional variant (mmDCGMs), which explore the strongly discriminative principle of max-margin learning to improve the predictive performance of DGMs in both supervised and semi-supervised learning, while retaining the generative capability. In semi-supervised learning, we use the predictions of a max-margin classifier as the missing labels instead of performing full posterior inference for efficiency; we also introduce additional max-margin and label-balance regularization terms of unlabeled data for effectiveness. We develop an efficient doubly stochastic subgradient algorithm for the piecewise linear objectives in different settings. Empirical results on various datasets demonstrate that: (1) max-margin learning can significantly improve the prediction performance of DGMs and meanwhile retain the generative ability; (2) in supervised learning, mmDGMs are competitive to the best fully discriminative networks when employing convolutional neural networks as the generative and recognition models; and (3) in semi-supervised learning, mmDCGMs can perform efficient inference and achieve state-of-the-art classification results on several benchmarks.

Authors

Jun Zhu

7 papers

Chongxuan Li

4 papers

Bo Zhang

1 papers

TL;DR

Abstract

Authors

References74 items

Statistical Learning Theory

Missing data

On the Quantitative Analysis of Decoder-Based Generative Models

Conditional Generative Moment-Matching Networks

Improved Techniques for Training GANs

Theano: A Python framework for fast computation of mathematical expressions

Learning to Generate with Memory

Auxiliary Deep Generative Models

Deep Residual Learning for Image Recognition

Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Importance Weighted Autoencoders

Semi-supervised Learning with Ladder Networks

Distributional Smoothing with Virtual Adversarial Training

Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks

Learning Deep Generative Models with Doubly Stochastic MCMC

Training generative neural networks via Maximum Mean Discrepancy optimization

Max-Margin Deep Generative Models

DRAW: A Recurrent Neural Network For Image Generation

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Generative Moment Matching Networks

Adam: A Method for Stochastic Optimization

Learning to generate chairs with convolutional neural networks

Deeply-Supervised Nets

Semi-supervised Learning with Deep Generative Models

Reweighted Wake-Sleep

Neural Variational Inference and Learning in Belief Networks

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Auto-Encoding Variational Bayes

Network In Network

Deep AutoRegressive Networks

Gibbs max-margin topic models with data augmentation

Improved Bayesian Logistic Supervised Topic Models with Data Augmentation

Deep Generative Stochastic Networks Trainable by Backprop

Deep Learning using Linear Support Vector Machines

Generalized Denoising Auto-Encoders as Generative Models

Maxout Networks

Stochastic Pooling for Regularization of Deep Convolutional Neural Networks

Large-Margin Predictive Latent Subspace Learning for Multiview Data Analysis

Theano: new features and speed improvements

Bayesian inference with posterior regularization and applications to infinite latent SVMs

Max-Margin Min-Entropy Models

Convolutional neural networks applied to house numbers digit classification

On deep generative models with applications to recognition

Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion

Introduction to Semi-Supervised Learning

Learning structural SVMs with latent variables

Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations

Deep Boltzmann Machines

Partially Observed Maximum Entropy Discrimination Markov Networks

Pegasos: primal estimated sub-gradient solver for SVM

Support vector machine learning for interdependent and structured output spaces

Learning methods for generic object recognition with invariance to pose and lighting

Max-Margin Markov Networks

Hidden Markov Support Vector Machines

Mean Field Theory for Sigmoid Belief Networks

Support-Vector Networks

The Laplacian Pyramid as a Compact Image Code

AUTO-ENCODING VARIATIONAL BAYES

GENERATIVE ADVERSARIAL NETS

Neural Networks for Machine Learning

Reweighted wakesleep . International Conference on Learning Representations

Dropout: a simple way to prevent neural networks from overfitting

MedLDA: maximum margin supervised topic models

Reading Digits in Natural Images with Unsupervised Feature Learning

2015 unsupervised feature learning

The Neural Autoregressive Distribution Estimator

Introduction to semisupervised learning. Synthesis lectures on artificial intelligence and machine learning

Visualizing Data using t-SNE

Stochastic Optimization

Gradient-based learning applied to document recognition

plug in the predictions of the unlabeled data ˆ y n and the groudtruth of the labeled data y n into the indicator functions in the label-balance regularization

on statistical machine learning, especially generative models for various learning tasks including unsupervised, (semi-)supervised and re-inforcement learning

the Department of Computer Science and Technology, Tsinghua University and a Fellow of Chinese Academy of Sciences, Beijing, China

Field of Study

Journal Information