A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks (2016-10-07T00:00:00.000000Z)

TL;DR

A simple baseline that utilizes probabilities from softmax distributions is presented, showing the effectiveness of this baseline across all computer vision, natural language processing, and automatic speech recognition, and it is shown the baseline can sometimes be surpassed.

Abstract

We consider the two related problems of detecting if an example is misclassified or out-of-distribution. We present a simple baseline that utilizes probabilities from softmax distributions. Correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified and out-of-distribution examples, allowing for their detection. We assess performance by defining several tasks in computer vision, natural language processing, and automatic speech recognition, showing the effectiveness of this baseline across all. We then show the baseline can sometimes be surpassed, demonstrating the room for future research on these underexplored detection tasks.

Authors

Dan Hendrycks

18 papers

Kevin Gimpel

12 papers

TL;DR

Abstract

Authors

References47 items

Adjusting for Dropout Variance in Batch Normalization and Weight Initialization

SGDR: Stochastic Gradient Descent with Warm Restarts

Bag of Tricks for Efficient Text Classification

Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units

Concrete Problems in AI Safety

Augmenting Supervised Neural Networks with Unsupervised Objectives for Large-scale Image Classification

Unsupervised Risk Estimation Using Only Conditional Independence Structure

Wide Residual Networks

Human-level concept learning through probabilistic program induction

THCHS-30 : A Free Chinese Speech Corpus

Posterior calibration and exploratory analysis for natural language processing models

Deep Unordered Composition Rivals Syntactic Methods for Text Classification

Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?

The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets

Adam: A Method for Stochastic Optimization

Explaining and Harnessing Adversarial Examples

Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters

An investigation of deep neural networks for noise robust speech recognition

The conference paper

Calibration of Confidence Measures in Speech Recognition

Learning Word Vectors for Sentiment Analysis

SUN database: Large-scale scene recognition from abbey to zoo

Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments

The relationship between Precision-Recall and ROC curves

An introduction to ROC analysis

RCV1: A New Benchmark Collection for Text Categorization Research

Mining and summarizing customer reviews

Thumbs up? Sentiment Classification using Machine Learning Techniques

Foundations of Statistical Natural Language Processing

The aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions

The Case against Accuracy Estimation for Comparing Induction Algorithms

Long Short-Term Memory

Confidence measures for hybrid HMM/ANN speech recognition

NewsWeeder: Learning to Filter Netnews

Building a Large Annotated Corpus of English: The Penn Treebank

Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS

Improving and generalizing weight initialization. arXiv, 2016b

Methods for detecting adversarial images and a colorful saliency map. arXiv

SGDR: Stochastic Gradient Descent with Restarts

TIMIT Acoustic-Phonetic Continuous Speech Corpus

Yaroslav Bulatov. notMNIST dataset

English Web Treebank

Learning Multiple Layers of Features from Tiny Images

Nancy Dahlgren, and Victor Zue. TIMIT Acoustic-Phonetic Continuous Speech Corpus. Linguistic Data Consortium

Connectionist Temporal Classiﬁcation: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

Connectionist Temporal Classiﬁcation: Labelling Unsegmented Sequences with Recurrent Neural Networks

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names