The Dual Information Bottleneck (2019-09-25T00:00:00.000000Z)

TL;DR

A theoretical analysis of the dualIB framework is provided, solving for the structure of its solutions, unraveling its superiority in optimizing the mean prediction error exponent and demonstrating its ability to preserve exponential forms of the original distribution.

Abstract

The Information Bottleneck (IB) framework is a general characterization of optimal representations obtained using a principled approach for balancing accuracy and complexity. Here we present a new framework, the Dual Information Bottleneck (dualIB), which resolves some of the known drawbacks of the IB. We provide a theoretical analysis of the dualIB framework; (i) solving for the structure of its solutions (ii) unraveling its superiority in optimizing the mean prediction error exponent and (iii) demonstrating its ability to preserve exponential forms of the original distribution. To approach large scale problems, we present a novel variational formulation of the dualIB for Deep Neural Networks. In experiments on several data-sets, we compare it to a variational form of the IB. This exposes superior Information Plane properties of the dualIB and its potential in improvement of the error.

Authors

Naftali Tishby

3 papers

Zoe Piran

1 papers

TL;DR

Abstract

Authors

References38 items

Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors

The Conditional Entropy Bottleneck

CEB Improves Model Robustness

Phase Transitions for the Information Bottleneck in Representation Learning

Direct Validation of the Information Bottleneck Principle for Deep Nets

Specializing Word Embeddings (for Parsing) by Information Bottleneck

Unpaired Image-to-Speech Synthesis With Multimodal Information Bottleneck

When Does Label Smoothing Help?

On Variational Bounds of Mutual Information

Divergence Functions in Information Geometry

Bregman Divergence Bounds and Universality Properties of the Logarithmic Loss

Bregman Divergence Bounds and the Universality of the Logarithmic Loss

Estimating Information Flow in Deep Neural Networks

Causal Deep Information Bottleneck

Inverting Supervised Representations with Autoregressive Neural Density Models

Emergence of Invariance and Disentanglement in Deep Representations

Opening the Black Box of Deep Neural Networks via Information

Information Dropout: Learning Optimal Representations Through Noisy Computation

Opening the Black Box

Wide Residual Networks

The Deterministic Information Bottleneck

Adam: A Method for Stochastic Optimization

Learning and generalization with the information bottleneck

Asymptotic Geometry of Multiple Hypothesis Testing

Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing)

Elements of Information Theory

Critical slowing down at a bifurcation

Multivariate Information Bottleneck

The information bottleneck method

Fundamentals of statistical exponential families: with applications in statistical decision theory

Information Theory and Statistical Mechanics

Deterministic annealing and the evolution of optimal information bottleneck representations

Deep Variational Information Bottleneck

Understanding Machine Learning From Theory to Algorithms 1st Edition Shwartz Solutions Manual

Cover and

An Information Theoretic Tradeoff between Complexity and Accuracy

Annealing and the Rate Distortion Problem

Information geometry and alternating minimization procedures

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names