OpenOOD: Benchmarking Generalized Out-of-Distribution Detection (2022-10-13T00:00:00.000000Z)

TL;DR

A unified, well-structured codebase called OpenOOD is built, which implements over 30 methods developed in relevant fields and provides a comprehensive benchmark under the recently proposed generalized OOD detection framework.

Abstract

Out-of-distribution (OOD) detection is vital to safety-critical machine learning applications and has thus been extensively studied, with a plethora of methods developed in the literature. However, the field currently lacks a unified, strictly formulated, and comprehensive benchmark, which often results in unfair comparisons and inconclusive results. From the problem setting perspective, OOD detection is closely related to neighboring fields including anomaly detection (AD), open set recognition (OSR), and model uncertainty, since methods developed for one domain are often applicable to each other. To help the community to improve the evaluation and advance, we build a unified, well-structured codebase called OpenOOD, which implements over 30 methods developed in relevant fields and provides a comprehensive benchmark under the recently proposed generalized OOD detection framework. With a comprehensive comparison of these methods, we are gratified that the field has progressed significantly over the past few years, where both preprocessing methods and the orthogonal post-hoc methods show strong potential.

Authors

Ziwei Liu

9 papers

Kaiyang Zhou

13 papers

Dan Hendrycks

18 papers

TL;DR

Abstract

Authors

References85 items

Enhancing Educational Insights Through Convolutional Neural Networks: A Comparative Analysis on CIFAR-10 and CIFAR-100 Datasets

ADBench: Anomaly Detection Benchmark

Emergent Abilities of Large Language Models

X-Risk Analysis for AI Research

Mitigating Neural Network Overconfidence with Logit Normalization

Simple Open-Vocabulary Object Detection with Vision Transformers

Out-of-distribution Detection with Deep Nearest Neighbors

Full-Spectrum Out-of-Distribution Detection

ViM: Out-Of-Distribution with Virtual-logit Matching

Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wild

Rethinking Reconstruction Autoencoder-Based Out-of-Distribution Detection

VOS: Learning What You Don't Know by Virtual Outlier Synthesis

Extreme Value Theory

PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures

ReAct: Out-of-distribution Detection With Rectified Activations

DICE: Leveraging Sparsification for Out-of-Distribution Detection

Generalized Out-of-Distribution Detection: A Survey

Open-Set Recognition: A Good Closed-Set Classifier is All You Need

On the Importance of Gradients for Detecting Distributional Shifts in the Wild

Unsolved Problems in ML Safety

Semantically Coherent Out-of-Distribution Detection

DRÆM – A discriminatively trained reconstruction embedding for surface anomaly detection

Energy-Based Open-World Uncertainty Modeling for Confidence Calibration

A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection

Towards Total Recall in Industrial Anomaly Detection

MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

CutPaste: Self-Supervised Learning for Anomaly Detection and Localization

Domain Generalization: A Survey

Adversarial Reciprocal Points Learning for Open Set Recognition

Open-Vocabulary Object Detection Using Captions

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Energy-based Out-of-distribution Detection

A Unifying Review of Deep and Shallow Anomaly Detection

CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances

Detecting Out-of-Distribution Examples with Gram Matrices

Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data

Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancy

Detecting semantic anomalies

Natural Adversarial Examples

A survey on Image Data Augmentation for Deep Learning

MNIST-C: A Robustness Benchmark for Computer Vision

MVTec AD — A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection

On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Why ReLU Networks Yield High-Confidence Predictions Far Away From the Training Data and How to Mitigate the Problem

Improving Reconstruction Autoencoder Out-of-distribution Detection with Mahalanobis Distance

Deep Anomaly Detection with Outlier Exposure

Open Set Learning with Counterfactual Images

A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks

Generative Probabilistic Novelty Detection with Adversarial Autoencoders

Latent Space Autoregression for Novelty Detection

Deep One-Class Classification

Places: A 10 Million Image Database for Scene Recognition

Predictive Uncertainty Estimation via Prior Networks

Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection

Learning Confidence for Out-of-Distribution Detection in Neural Networks

Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Generative OpenMax for Multi-Class Open Set Classification

On Calibration of Modern Neural Networks

Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks

Deep Residual Learning for Image Recognition

Towards Open Set Deep Networks

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Probability Models for Open Set Recognition

Toward Open Set Recognition

ImageNet classification with deep convolutional neural networks

The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]

Anomaly detection: A survey

Isolation Forest

80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition

The Black Swan: The Impact of the Highly Improbable

Multiple Classifier Systems