In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation (2023-06-01T00:00:00.000000Z)

TL;DR

A novel test OOD dataset is introduced, each sample checked to be ID free, which with its fine-grained range of OOD classes allows for a detailed analysis of an OOD detector's strengths and failure modes, particularly when paired with a number of synthetic"OOD unit-tests".

Abstract

Out-of-distribution (OOD) detection is the problem of identifying inputs which are unrelated to the in-distribution task. The OOD detection performance when the in-distribution (ID) is ImageNet-1K is commonly being tested on a small range of test OOD datasets. We find that most of the currently used test OOD datasets, including datasets from the open set recognition (OSR) literature, have severe issues: In some cases more than 50$\%$ of the dataset contains objects belonging to one of the ID classes. These erroneous samples heavily distort the evaluation of OOD detectors. As a solution, we introduce with NINCO a novel test OOD dataset, each sample checked to be ID free, which with its fine-grained range of OOD classes allows for a detailed analysis of an OOD detector's strengths and failure modes, particularly when paired with a number of synthetic"OOD unit-tests". We provide detailed evaluations across a large set of architectures and OOD detection methods on NINCO and the unit-tests, revealing new insights about model weaknesses and the effects of pretraining on OOD detection performance. We provide code and data at https://github.com/j-cb/NINCO.

Authors

Julian Bitterwolf

1 papers

Maximilian Mueller

1 papers

Matthias Hein

2 papers

TL;DR

Abstract

Authors

References51 items

A framework for benchmarking class-out-of-distribution detection and its application to ImageNet

Delving into Out-of-Distribution Detection with Vision-Language Representations

OpenOOD: Benchmarking Generalized Out-of-Distribution Detection

RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection

On the Usefulness of Deep Ensemble Diversity for Out-of-Distribution Detection

Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition

Breaking Down Out-of-Distribution Detection: Many Methods Based on OOD Training Data Estimate a Combination of the Same Core Quantities

When does dough become a bagel? Analyzing the remaining mistakes on ImageNet

DeiT III: Revenge of the ViT

Out-of-distribution Detection with Deep Nearest Neighbors

ViM: Out-Of-Distribution with Virtual-logit Matching

ReAct: Out-of-distribution Detection With Rectified Activations

A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges

Open-Set Recognition: A Good Closed-Set Classifier is All You Need

Can multi-label classification networks know what they don't know?

OODformer: Out-Of-Distribution Detection Transformer

How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection

Provably Adversarially Robust Detection of Out-of-Distribution Data (Almost) for Free

Exploring the Limits of Out-of-Distribution Detection

MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space

Learning Transferable Visual Models From Natural Language Supervision

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Energy-based Out-of-distribution Detection

MYNursingHome: A fully-labelled image dataset for indoor object classification.

CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances

Certifiably Adversarially Robust Detection of Out-of-Distribution Data

Evaluating Machine Accuracy on ImageNet

Big Transfer (BiT): General Visual Representation Learning

Self-Training With Noisy Student Improves ImageNet Classification

Natural Adversarial Examples

Why ReLU Networks Yield High-Confidence Predictions Far Away From the Training Data and How to Mitigate the Problem

Deep Anomaly Detection with Outlier Exposure

A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks

Places: A 10 Million Image Database for Scene Recognition

The iNaturalist Species Classification and Detection Dataset

A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks

Towards Open Set Deep Networks

Food-101 - Mining Discriminative Components with Random Forests

ImageNet Large Scale Visual Recognition Challenge

Describing Textures in the Wild

ImageNet: A large-scale hierarchical image database

Scaling Out-of-Distribution Detection for Real-World Settings

A Simple Parametric Classification Baseline for Generalized Category Discovery

Hyperparameter-Free Out-of-Distribution Detection Using Cosine Similarity

Pytorch image models

Openimages: A public dataset for large-scale multi-label and multi-class image classification

Learning Multiple Layers of Features from Tiny Images

Book Reviews: WordNet: An Electronic Lexical Database

ViT-B-384-oai 56.0 51.8 −4 54.6 −1 53.6 −2 40.9 −15 39.8 −16 36.9 −19 50.4 −6 36.6 −19 33.8 −22 34.1 −22 clip 86.6 ViT-B-384-l2b 65

2 KL-M 72. method Pl-f Pl-c Spc-f Spc-c IN-f IN-c txt-f txt-43 txt-c OpO-f OpO-c iNat-f iNat-c IN1K-f IN1K-c

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names