Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020-02-20T00:00:00.000000Z)

TL;DR

It is shown that deep ensembles provide an effective mechanism for approximate Bayesian marginalization, and a related approach is proposed that further improves the predictive distribution by marginalizing within basins of attraction, without significant overhead.

Abstract

The key distinguishing property of a Bayesian approach is marginalization, rather than using a single setting of weights. Bayesian marginalization can particularly improve the accuracy and calibration of modern deep neural networks, which are typically underspecified by the data, and can represent many compelling but different solutions. We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization, and propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction, without significant overhead. We also investigate the prior over functions implied by a vague distribution over neural network weights, explaining the generalization properties of such models from a probabilistic perspective. From this perspective, we explain results that have been presented as mysterious and distinct to neural network generalization, such as the ability to fit images with random labels, and show that these results can be reproduced with Gaussian processes. We also show that Bayesian model averaging alleviates double descent, resulting in monotonic performance improvements with increased flexibility. Finally, we provide a Bayesian perspective on tempering for calibrating predictive distributions.

Authors

A. Wilson

10 papers

Pavel Izmailov

4 papers

TL;DR

Abstract

Authors

References97 items

On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification

What Are Bayesian Neural Network Posteriors Really Like?

Optimal Regularization Can Mitigate Double Descent

Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning

How Good is the Bayes Posterior in Deep Neural Networks Really?

The Case for Bayesian Deep Learning

Learning under Model Misspecification: Applications to Variational and Ensemble methods

Fantastic Generalization Measures and Where to Find Them

Deep double descent: where bigger models and more data hurt

Safe-Bayesian Generalized Linear Regression

Introducing an Explicit Symplectic Integration Scheme for Riemannian Manifold Hamiltonian Monte Carlo

Deep Ensembles: A Loss Landscape Perspective

SciPy 1.0: fundamental algorithms for scientific computing in Python

Subspace Inference for Bayesian Deep Learning

The Functional Neural Process

Large Scale Structure of Neural Network Loss Landscapes

Understanding Generalization through Visualizations

Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision

Output-Constrained Bayesian Neural Networks

Functional Variational Bayesian Neural Networks

A Simple Baseline for Bayesian Uncertainty in Deep Learning

A Primer on PAC-Bayesian Learning

Calibrated Bayes factors for model comparison

Reconciling modern machine-learning practice and the classical bias–variance trade-off

Uncertainty in Neural Networks: Approximately Bayesian Ensembling

Uncertainty in Neural Networks: Bayesian Ensembling

GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration

Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam

Averaging Weights Leads to Wider Optima and Better Generalization

Essentially No Barriers in Neural Network Energy Landscape

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

A Scalable Laplace Approximation for Neural Networks

Deep Image Prior

A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks

Exploring Generalization in Deep Learning

On Calibration of Modern Neural Networks

Emergence of Invariance and Disentanglement in Deep Representations

Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data

What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

Sharp Minima Can Generalize For Deep Nets

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Entropy-SGD: biasing gradient descent into wide valleys

Understanding deep learning requires rethinking generalization

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Deep Residual Learning for Image Recognition

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Weight Uncertainty in Neural Networks

Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing It

Very Deep Convolutional Networks for Large-Scale Image Recognition

Stochastic Gradient Descent

A general framework for updating belief distributions

The Safe Bayesian - Learning the Learning Rate via the Mixability Gap

Bayesian Learning via Stochastic Gradient Langevin Dynamics

Adaptive and Learning Systems for Signal Processing, Communication, and Control

Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations

Rademacher Complexity Bounds for Non-I.I.D. Processes

Adaptive and Learning Systems for Signal Processing, Communications, and Control

Pattern Recognition and Machine Learning

Information-theoretic upper and lower bounds for statistical estimation

Gaussian Processes For Machine Learning

Expectation Propagation for approximate Bayesian inference

(Not) Bounding the True Error

PAC-Bayesian model averaging

Keeping the neural networks simple by minimizing the description length of the weights

Inference from Iterative Simulation Using Multiple Sequences

Minimum complexity density estimation

On the ability of the optimal perceptron to generalise

Occam's razor

Predictions from prior samples

2020), which show great improvements over standard SGD training, are in fact very encouraging of deriving inspiration from Bayesian procedures in deep learning

2019) we use ResNet-18 on CIFAR-100; we consider original labels, 10% and 20% label corruption. For networks of every width we reuse the hyper-paramerers used for PreResNet

We use diagonal Laplace approximation to compute an estimate of marginal likelihood for each level of label corruption

2019) and show the results