Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (2020-06-01T00:00:00.000000Z)

TL;DR

Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs, by adding a weight normalization step during training and replacing the output layer with a Gaussian process and outperforms the other single-model approaches.

Abstract

Bayesian neural networks (BNN) and deep ensembles are principled approaches to estimate the predictive uncertainty of a deep learning model. However their practicality in real-time, industrial-scale applications are limited due to their heavy memory and inference cost. This motivates us to study principled approaches to high-quality uncertainty estimation that require only a single deep neural network (DNN). By formalizing the uncertainty quantification as a minimax learning problem, we first identify input distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data in the input space, as a necessary condition for a DNN to achieve high-quality (i.e., minimax optimal) uncertainty estimation. We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs, by adding a weight normalization step during training and replacing the output layer with a Gaussian process. On a suite of vision and language understanding tasks and on modern architectures (Wide-ResNet and BERT), SNGP is competitive with deep ensembles in prediction, calibration and out-of-domain detection, and outperforms the other single-model approaches.

Authors

Balaji Lakshminarayanan

8 papers

Dustin Tran

6 papers

Zi Lin

4 papers

TL;DR

Abstract

Authors

References93 items

Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors

User Utterance Acquisition for Training Task-Oriented Bots: A Review of Challenges, Techniques and Opportunities

Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network

Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

Towards neural networks that provably know when they don't know

Verified Uncertainty Calibration

Out-of-Domain Detection for Natural Language Understanding in Dialog Systems

An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

Isotropic Maximization Loss and Entropic Score: Fast, Accurate, Scalable, Unexposed, Turnkey, and Native Neural Networks Out-of-Distribution Detection.

Distinction Maximization Loss: Fast, Scalable, Turnkey, and Native Neural Networks Out-of-Distribution Detection simply by Replacing the SoftMax Loss

Approximate Inference Turns Deep Networks into Gaussian Processes

Residual Networks as Flows of Diffeomorphisms

Towards Open Intent Discovery for Conversational Text

A deterministic and computable Bernstein-von Mises theorem

Measuring Calibration in Deep Learning

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness

Using Pre-Training Can Improve Model Robustness and Uncertainty

Why ReLU Networks Yield High-Confidence Predictions Far Away From the Training Data and How to Mitigate the Problem

Single-Model Uncertainties for Deep Learning

Excessive Invariance Causes Adversarial Vulnerability

Invertible Residual Networks

Principled Uncertainty Estimation for Deep Neural Networks

Sorting out Lipschitz function approximation

Prior Networks for Detection of Adversarial Attacks

Challenges for Toxic Comment Classification: An In-Depth Error Analysis

Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors

Evidential Deep Learning to Quantify Classification Uncertainty

Calibrating Deep Convolutional Gaussian Processes

Reachability Analysis of Deep Neural Networks with Provable Guarantees

Representing smooth functions as compositions of near-identity functions with implications for deep network optimization

Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection

Regularisation of neural networks by enforcing Lipschitz continuity

Predictive Uncertainty Estimation via Prior Networks

Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling

Spectral Normalization for Generative Adversarial Networks

A Scalable Laplace Approximation for Neural Networks

i-RevNet: Deep Invertible Networks

Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach

Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples

DOC: Deep Open Classification of Text Documents

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks

On Calibration of Modern Neural Networks

Improved Training of Wasserstein GANs

Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Stochastic Variational Deep Kernel Learning

Density estimation using Real NVP

Robust Large Margin Deep Neural Networks

Wide Residual Networks

Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)

Towards Open Set Deep Networks

Probabilism, entropies and strictly proper scoring rules

How Can Deep Rectifier Networks Achieve Linear Separability and Preserve Distances?

Scalable Bayesian Optimization Using Deep Neural Networks

NICE: Non-linear Independent Components Estimation

Probability Models for Open Set Recognition

Manifold Gaussian Processes for regression

Finite Sample Bernstein -- von Mises Theorem for Semiparametric Problems

Proper local scoring rules

Learning Non-Linear Combinations of Kernels

Reliability, Sufficiency, and the Decomposition of Proper Scores

Classification with a Reject Option using a Hinge Loss

Random Features for Large-Scale Kernel Machines

Probabilistic forecasts, calibration and sharpness

Strictly Proper Scoring Rules, Prediction, and Estimation

Local distance preservation in the GP-LVM through back constraints

Random Projection, Margins, Kernels, and Feature-Selection

Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory

Gaussian Processes in Machine Learning

On the Bernstein-von Mises Theorem with Infinite Dimensional Parameters

Statistical Decision Theory and Bayesian Analysis, Second Edition

A Practical Bayesian Framework for Backpropagation Networks