Classifier Calibration: With Application to Threat Scores in Cybersecurity (2021-02-09T00:00:00.000000Z)

TL;DR

This article proposes expanding these classifiers’ scores to higher dimensions to boost the calibrator’s performance, and suggests multi-score calibration provides better performance than single score calibration in the majority of experiments, including the two real datasets.

Abstract

This article explores the calibration of a classifier output score in binary classification problems. A calibrator is a function that maps the arbitrary classifier score, of a testing observation, onto [0,1] to provide an estimate for the posterior probability of belonging to one of the two classes. Calibration is important for two reasons; first, it provides a meaningful score, that is the posterior probability; second, it puts the scores of different classifiers on the same scale for comparable interpretation. The article presents three main contributions: (1) Introducing multi-score calibration, when more than one classifier provides a score for a single observation. (2) Introducing the exact analogy between two scenarios: (a) designing a classifier from a set of features, and (b) designing a calibrator, to generate a single calibrated score, from a set of scores of different classifiers. Hence, we propose expanding these classifiers’ scores to higher dimensions to boost the calibrator’s performance. (3) Conducting a massive simulation study, in the order of 24,000 experiments, that incorporates different configurations, in addition to experimenting on three real datasets from the cybersecurity domain. The results show that there is no overall winner among the different calibrators and different configurations. However, general advices for practitioners include the following: the Platt’s calibrator (J. Platt et al., 1999), a version of the logistic regression that decreases bias for a small sample size, has a very stable and acceptable performance among all experiments; our suggested multi-score calibration provides better performance than single score calibration in the majority of experiments, including the two real datasets. In addition, expanding the scores can help in some experiments.

References44 items

The elements of statistical learning: data mining, inference, and prediction, 2nd Edition

Prudence When Assuming Normality: an advice for machine learning practitioners

Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection

Scalable cyber-security analytics with a new summary-based approximate query engine

Calibration of medical diagnostic classifier scores to the probability of disease

Investigation of methods for calibration of classifier scores to probability of disease

A classifier fusion method based on classifier accuracy

Assessing classifiers in terms of the partial area under the ROC curve

Mobile-sandbox: having a deeper look into android applications

Classifier variability: Accounting for training and testing

Semiparametric estimation of the relationship between ROC operating points and the test-result scale: application to the proper binormal model.

Scikit-learn: Machine Learning in Python

Nonparametric estimation of the threshold at an operating point on the ROC curve

Reliable and computationally efficient maximum-likelihood estimation of "proper" binormal ROC curves.

Online supervised spam filter evaluation

Assessing Classifiers from Two Independent Data Sets Using ROC Analysis: A Nonparametric Approach

Predicting good probabilities with supervised learning

The Elements of Statistical Learning: Data Mining, Inference, and Prediction

Transforming classifier scores into accurate multiclass probability estimates

A Theoretical Study on Six Classifier Fusion Strategies

Learning and making decisions when costs and probabilities are both unknown

"Proper" Binormal ROC Curves: Theory and Maximum-Likelihood Estimation.

Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data.

On Combining Classifiers

The "proper" binormal model: parametric receiver operating characteristic curve estimation with degenerate data.

Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation.

Proper receiver operating characteristic analysis: the bigamma model.

Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method.

Introduction to the Theory of Nonparametric Statistics

A Probability Distribution and its Uses in Fitting Data

Emerging

Theory of rank tests

"PROOF"

For more information on this or any other computing topic

DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket

TREC 2007 Spam Track Overview

on Pattern Analysis and Machine Intelligence

Predicting the Probability of Correct Classification ; CU-CS-974-04

Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods

VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY

For every threshold value θ ∈ R , with the corresponding probabilities F X ( θ ) and F Y ( θ )

Because the mapping p is strictly monotonically increasing, it is a bijection

These two properties are not satisfied with relaxing the strictly monotonic assumption and considering p a mono-tonic increasing function

Because the function p is a bijection, its inverse p − 1 is as well