Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning

Published in

International Conference on Learning Representa...(2020)

External Links:

Generate Graph

TL;DR

This work focuses on in-domain uncertainty for image classification and introduces the deep ensemble equivalent (DEE) and shows that many sophisticated ensembling techniques are equivalent to an ensemble of very few independently trained networks in terms of the test log-likelihood.

Abstract

Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance. At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation. In this work, we focus on in-domain uncertainty for image classification. We explore the standards for its quantification and point out pitfalls of existing metrics. Avoiding these pitfalls, we perform a broad study of different ensembling techniques. To provide more insight in the broad comparison, we introduce the deep ensemble equivalent (DEE) and show that many sophisticated ensembling techniques are equivalent to an ensemble of very few independently trained networks in terms of the test log-likelihood.

Authors

Arsenii Ashukha

3 papers

Alexander Lyzhov

1 papers

Dmitry Molchanov

2 papers

References60 items

Hydra: Preserving Ensemble Diversity for Model Distillation

Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over the Simplex

Verified Uncertainty Calibration

Bayesian Inference for Large Scale Image Classification

Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning

Published in

International Conference on Learning Representa...(2020)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Arsenii Ashukha

3 papers

Alexander Lyzhov

1 papers

Dmitry Molchanov

2 papers

References60 items

Hydra: Preserving Ensemble Diversity for Model Distillation

Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over the Simplex

Verified Uncertainty Calibration

Bayesian Inference for Large Scale Image Classification

Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision

Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over Simplex

Measuring Calibration in Deep Learning

Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off

Evaluation of Neural Network Uncertainty Estimation with Application to Resource-Constrained Platforms

Evaluating model calibration in classification

Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

A Simple Baseline for Bayesian Uncertainty in Deep Learning

Deterministic Variational Inference for Robust Bayesian Neural Networks

Inhibited Softmax for Uncertainty Estimation in Neural Networks

Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

Adversarial Distillation of Bayesian Neural Network Posteriors

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Averaging Weights Leads to Wider Optima and Better Generalization

Predictive Uncertainty Estimation via Prior Networks

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Super-convergence: very fast training of neural networks using large learning rates

A Scalable Laplace Approximation for Neural Networks

Bayesian Uncertainty Estimation for Batch Normalized Deep Networks

Uncertainty Estimation via Stochastic Batch Normalization

Automatic differentiation in PyTorch

mixup: Beyond Empirical Risk Minimization

On Calibration of Modern Neural Networks

Snapshot Ensembles: Train 1, get M for free

Multiplicative Normalizing Flows for Variational Bayesian Neural Networks

Soft Weight-Sharing for Neural Network Compression

Variational Dropout Sparsifies Deep Neural Networks

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Wide Residual Networks

How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

Deep Residual Learning for Image Recognition

Bayesian dark knowledge

Variational Dropout and the Local Reparameterization Trick

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Weight Uncertainty in Neural Networks

Optimizing Neural Networks with Kronecker-factored Approximate Curvature

Distilling the Knowledge in a Neural Network

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Obtaining Well Calibrated Probabilities Using Bayesian Binning

Going deeper with convolutions

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Intriguing properties of neural networks

Fast dropout training

Bayesian Learning via Stochastic Gradient Langevin Dynamics

Neural network with ensembles

Accuracy-Rejection Curves (ARCs) for Comparing Classification Methods with a Reject Option

Evaluating Predictive Uncertainty Challenge

Neural Network Ensembles

Uncertainty in Deep Learning

92.45 on cifar-10

Dropout: a simple way to prevent neural networks from overfitting

Learning Multiple Layers of Features from Tiny Images

VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY

Field of Study

Computer ScienceMathematics

Journal Information

Name

ArXiv

Volume

abs/2002.06470

Venue Information

Name

International Conference on Learning Representations

Type

conference

URL

https://iclr.cc/

Alternate Names

Int Conf Learn Represent
ICLR

TL;DR

Abstract

Authors

References60 items

Hydra: Preserving Ensemble Diversity for Model Distillation

Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over the Simplex

Verified Uncertainty Calibration

Bayesian Inference for Large Scale Image Classification

Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

TL;DR

Abstract

Authors

References60 items

Hydra: Preserving Ensemble Diversity for Model Distillation

Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over the Simplex

Verified Uncertainty Calibration

Bayesian Inference for Large Scale Image Classification

Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision

Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over Simplex

Measuring Calibration in Deep Learning

Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off

Evaluation of Neural Network Uncertainty Estimation with Application to Resource-Constrained Platforms

Evaluating model calibration in classification

Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

A Simple Baseline for Bayesian Uncertainty in Deep Learning

Deterministic Variational Inference for Robust Bayesian Neural Networks

Inhibited Softmax for Uncertainty Estimation in Neural Networks

Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

Adversarial Distillation of Bayesian Neural Network Posteriors

Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Averaging Weights Leads to Wider Optima and Better Generalization

Predictive Uncertainty Estimation via Prior Networks

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Super-convergence: very fast training of neural networks using large learning rates

A Scalable Laplace Approximation for Neural Networks

Bayesian Uncertainty Estimation for Batch Normalized Deep Networks

Uncertainty Estimation via Stochastic Batch Normalization

Automatic differentiation in PyTorch

mixup: Beyond Empirical Risk Minimization

On Calibration of Modern Neural Networks

Snapshot Ensembles: Train 1, get M for free

Multiplicative Normalizing Flows for Variational Bayesian Neural Networks

Soft Weight-Sharing for Neural Network Compression

Variational Dropout Sparsifies Deep Neural Networks

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Wide Residual Networks

How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

Deep Residual Learning for Image Recognition

Bayesian dark knowledge

Variational Dropout and the Local Reparameterization Trick

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Weight Uncertainty in Neural Networks

Optimizing Neural Networks with Kronecker-factored Approximate Curvature

Distilling the Knowledge in a Neural Network

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Obtaining Well Calibrated Probabilities Using Bayesian Binning

Going deeper with convolutions

Very Deep Convolutional Networks for Large-Scale Image Recognition

ImageNet Large Scale Visual Recognition Challenge

Intriguing properties of neural networks

Fast dropout training

Active Learning

Bayesian Learning via Stochastic Gradient Langevin Dynamics

Neural network with ensembles

Accuracy-Rejection Curves (ARCs) for Comparing Classification Methods with a Reject Option

Evaluating Predictive Uncertainty Challenge

Neural Network Ensembles

Uncertainty in Deep Learning

92.45 on cifar-10

Dropout: a simple way to prevent neural networks from overfitting

Learning Multiple Layers of Features from Tiny Images

VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type