Deep Variational Information Bottleneck (2017-01-01T00:00:00.000000Z)

TL;DR

It is shown that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.

Abstract

We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method "Deep Variational Information Bottleneck", or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack.

Authors

Joshua V. Dillon

4 papers

Alexander A. Alemi

5 papers

Ian Fischer

1 papers

TL;DR

Abstract

Authors

References49 items

Information Dropout: Learning Optimal Representations Through Noisy Computation

beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

Universal Adversarial Perturbations

Deep Variational Canonical Correlation Analysis

Robustness of classifiers: from adversarial to random noise

Towards Evaluating the Robustness of Neural Networks

Differential Privacy as a Mutual Information Constraint

Adversarial examples in the physical world

On the Relation Between Identifiability, Differential Privacy, and Mutual-Information Privacy

Early Visual Concept Learning with Unsupervised Deep Learning

Relevant sparse codes with variational information bottleneck

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

The Limitations of Deep Learning in Adversarial Settings

Adversarial Manipulation of Deep Representations

DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks

Learning with a Strong Adversary

The Variational Fair Autoencoder

Confusing Deep Convolution Networks by Relabelling

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

The Virtues of Peer Pressure: A Simple Method for Discovering High-Value Mistakes

Weight Uncertainty in Neural Network

Weight Uncertainty in Neural Networks

Deep learning and the information bottleneck principle

Adam: A Method for Stochastic Optimization

Explaining and Harnessing Adversarial Examples

Towards Open World Recognition

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

On the relation between identifiability, differential privacy, and mutual-information privacy

Auto-Encoding Variational Bayes

Intriguing properties of neural networks

Predictive information in a sensory population

The conference paper

Understanding the difficulty of training deep feedforward neural networks

ImageNet: A large-scale hierarchical image database

Learning and generalization with the information bottleneck

Information-based clustering.

Information Bottleneck for Gaussian Variables

The IM algorithm: a variational approach to Information Maximization

How Many Clusters? An Information-Theoretic Perspective

The information bottleneck method

Acceleration of stochastic approximation by averaging

Under review as a conference paper at ICLR 2020 many domain adaptation methods

Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS

Regularizing neural networks by penalizing conﬁdent output predictions

Multivariate sharp quadratic bounds via -strong convexity and the Fenchel connection

Dale Schuurmans, and Csaba Szepesvári. Learning with a strong adversary . CoRR, abs/1511

Information Theory, Inference, and Learning Algorithms

Predictability , Complexity , and Learning

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names