Machine learning with a reject option: a survey

Published in

Machine-mediated learning(2021)

External Links:

Generate Graph DownloadPDF

TL;DR

The conditions leading to two types of rejection, ambiguity and novelty rejection, are introduced and the existing architectures for models with rejection are defined and the standard techniques for learning such models are described.

Abstract

Machine learning models always make a prediction, even when it is likely to be inaccurate. This behavior should be avoided in many decision support applications, where mistakes can have severe consequences. Albeit already studied in 1970, machine learning with rejection recently gained interest. This machine learning subfield enables machine learning models to abstain from making a prediction when likely to make a mistake. This survey aims to provide an overview on machine learning with rejection. We introduce the conditions leading to two types of rejection, ambiguity and novelty rejection, which we carefully formalize. Moreover, we review and categorize strategies to evaluate a model’s predictive and rejective quality. Additionally, we define the existing architectures for models with rejection and describe the standard techniques for learning such models. Finally, we provide examples of relevant application domains and show how machine learning with rejection relates to other machine learning research areas.

Authors

Kilian Hendrickx

1 papers

Lorenzo Perini

1 papers

Dries Van der Plas

1 papers

References277 items

Can We Trust Fair-AI?

A Model-Agnostic Heuristics for Selective Classification

Sources of Uncertainty in Supervised Machine Learning -- A Statisticians' View

How to Allocate your Label Budget? Choosing between Active Learning and Learning to Reject in Anomaly Detection

Estimating the Contamination Factor's Distribution in Unsupervised Anomaly Detection

Machine learning with a reject option: a survey

Published in

Machine-mediated learning(2021)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Kilian Hendrickx

1 papers

Lorenzo Perini

1 papers

Dries Van der Plas

1 papers

References277 items

Can We Trust Fair-AI?

A Model-Agnostic Heuristics for Selective Classification

Sources of Uncertainty in Supervised Machine Learning -- A Statisticians' View

How to Allocate your Label Budget? Choosing between Active Learning and Learning to Reject in Anomaly Detection

Estimating the Contamination Factor's Distribution in Unsupervised Anomaly Detection

Wannes Meert

1 papers

Jesse Davis

2 papers

AUC-based Selective Classification

Transferring the Contamination Factor between Anomaly Detection Domains by Shape Similarity

A meta-learning BCI for estimating decision confidence

Towards Better Selective Classification

Nonparametric Uncertainty Quantification for Single Deterministic Neural Network

Uncertainty-Based Rejection in Machine Learning: Implications for Model Development and Interpretability

Management of validation of HPLC method for determination of acetylsalicylic acid impurities in a new pharmaceutical product

Multilabel Classification with Partial Abstention: Bayes-Optimal Prediction under Label Independence

Online Selective Classification with Limited Feedback

Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis

Bayesian Active Meta-Learning for Reliable and Efficient AI-Based Demodulation

Securing Deep Learning Models with Autoencoder based Anomaly Detection

Two Coupled Rejection Metrics Can Tell Adversarial Examples Apart

Probabilistic personalised cascade with abstention

SCRIB: Set-classifier with Class-specific Risk Bounds for Blackbox Models

Exponential Savings in Agnostic Active Learning Through Abstention

Optimal strategies for reject option classifiers

How to define a rejection class based on model learning?

Second opinion needed: communicating uncertainty in medical machine learning

Fast and Accurate $k$-means++ via Rejection Sampling

Confidence Estimation via Auxiliary Models

Know your limits: Uncertainty estimation with ReLU classifiers fails at reliable OOD detection

A comprehensive survey and analysis of generative models in machine learning

Versatile Verification of Tree Ensembles

ATRO: Adversarial Training with a Rejection Option

Classification with Rejection Based on Cost-sensitive Classification

Selective Classification via One-Sided Prediction

Probability of default estimation, with a reject option

Risk-Controlled Selective Prediction for Regression Deep Neural Network Models

Generalized Neural Framework for Learning with Rejection

Classification Under Human Assistance

Consistent Estimators for Learning to Defer to an Expert

Bounded–abstaining classification for breast tumors in imbalanced ultrasound images

Active Learning for Classification with Abstention

Uncertainty-Based Rejection Wrappers for Black-Box Classifiers

A Novel Meta Learning Framework for Feature Selection using Data Synthesis and Fuzzy Similarity

AUC Optimization with a Reject Option

Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data

Self-Adaptive Training: beyond Empirical Risk Minimization

Asymmetric Distribution Measure for Few-shot Learning

Anomaly Detection

Arbitrated Dynamic Ensemble with Abstaining for Time-Series Forecasting on Data Streams

Performance visualization spaces for classification with rejection option

Active Learning with Abstaining Classifiers for Imbalanced Drifting Data Streams

Aleatoric and Epistemic Uncertainty in Machine Learning: A Tutorial Introduction

Deep neural rejection against adversarial examples

Epistemic Uncertainty Sampling

Deep Gamblers: Learning to Abstain with Portfolio Theory

Combating Label Noise in Deep Learning Using Abstention

On discriminative learning of prediction uncertainty

Binary Classification with Bounded Abstention Rate

Reliable Multi-label Classification: Prediction with Partial Abstention

Interpretable Cascade Classifiers with Abstention

Hybrid Models with Deep and Invertible Features

On the Calibration of Multiclass Classification with Rejection

SelectiveNet: A Deep Neural Network with an Integrated Reject Option

Semi-Supervised Anomaly Detection with an Application to Water Analytics

Meta-Learning: A Survey

Mitigating Concept Drift via Rejection

Rotation-blended CNNs on a New Open Dataset for Tropical Cyclone Image-to-intensity Regression

Uncertainty-Aware Attention for Reliable Interpretation and Prediction

COBRAS-TS: A new approach to Semi-Supervised Clustering of Time Series

To Trust Or Not To Trust A Classifier

Interpretable machine learning with reject option

Online ensemble learning with abstaining classifiers for drifting and noisy data streams

Machine Learning with Abstention for Automated Liver Disease Diagnosis

Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer

SafePredict: A Meta-Algorithm for Machine Learning That Uses Refusals to Guarantee Correctness

Probabilistic extension and reject options for pairwise LVQ

Twin SVM with a reject option through ROC curve

Selective Classification for Deep Neural Networks

Blocking Transferability of Adversarial Examples in Black-Box Learning Systems

Online Learning with Abstention

On Reject and Refine Options in Multicategory Classification

ROC-based cost-sensitive classification with a reject option

Classification with reject option for software defect prediction

Background Check: A General Technique to Build More Reliable and Versatile Classifiers

Optimal local rejection for classifiers

Reliable prediction of anti-diabetic drug failure using a reject option

Learning with Rejection

Local Reject Option for Deterministic Multi-class SVM

Evidence-based uncertainty sampling for active learning

Learning to Abstain from Binary Prediction

Efficient rejection strategies for prototype-based classification

Datasets meta-feature description for recommending feature selection algorithm

Robust classification with reject option using the self-organizing map

Classification with a reject option under Concept Drift: The Droplets algorithm

Image Classification with Rejection using Contextual Information

Learning Vector Quantization with Adaptive Cost-Based Outlier-Rejection

What Is Machine Learning

Consistency of plug-in confidence sets for classification in semi-supervised learning

Siamese neural network based similarity metric for inertial gesture classification and rejection

Robust hyperspectral image classification with rejection fields

100

Performance measures for classification systems with rejection

101

Optimum Reject Options for Prototype-based Classification

102

Classification with confidence

103

Classification with Reject Option Using the Self-Organizing Map

104

Local Rejection Strategies for Learning Vector Quantization

105

Combination of One-Class Support Vector Machines for Classification with Reject Option

106

Random Forest for Reliable Pre-classification of Handwritten Characters

107

Beyond Disagreement-Based Agnostic Active Learning

108

Classification with rejection based on various SVM techniques

109

A review of novelty detection

110

Highly Accurate Recognition of Human Postures and Activities Through Classification With Rejection

111

Rejection Schemes in Multi-class Classification -- Application to Handwritten Character Recognition

112

Multi-label classification with a reject option

113

Classifiers with a reject option for early time-series classification

114

Classification with reject option using contextual information

115

Reliable Classification of Vehicle Types Based on Cascade Classifier Ensembles

116

Weighted extreme learning machine for imbalance learning

117

MetaFraud: A Meta-Learning Framework for Detecting Financial Fraud

118

Multiple classifier combination using reject options and markov fusion networks

119

An Ensemble of Rejecting Classifiers for Anomaly Detection of Audio Events

120

Heartbeat Classification Using Support Vector Machines (SVMs) with an Embedded Reject Option

121

Design of reject rules for ECOC classification systems

122

Agnostic Selective Classification

123

A Classification Approach with a Reject Option for Multi-label Problems

124

A risk bound for ensemble classification with a reject option

125

Diagnostic of Pathology on the Vertebral Column with Embedded Reject Option

126

A Rejection Option for the Multilayer Perceptron Using Hyperplanes

127

Trading off Mistakes and Don't-Know Predictions

128

An Introduction to Conditional Random Fields

129

The data replication method for the classification with reject option

130

Adapting cost-sensitive learning for reject option

131

An Optimum Class-Rejective Decision Rule and Its Evaluation

132

On the Foundations of Noise-free Selective Classification

133

Classification Methods with Reject Option Based on Convex Risk Minimization

134

An Ordinal Data Method for the Classification with Reject Option

135

Accuracy-Rejection Curves (ARCs) for Comparing Classification Methods with a Reject Option

136

Isolation Forest

137

Pattern rejection strategies for the design of self-paced EEG-based Brain-Computer Interfaces

138

Support Vector Machines with a Reject Option

139

Multi-level Classification of Emphysema in HRCT Lung Images Using Delegated Classifiers

140

Classification with reject option in gene expression data

141

General solution and learning method for binary classification with performance constraints

142

Growing a multi-class classifier with a reject option

143

Classification with a Reject Option using a Hinge Loss

144

A Kernel Based Rejection Method for Supervised Classification

145

Learning to Classify Ordinal Data: The Data Replication Method

146

A Novel Classification-Rejection Sphere SVMs for Multi-class Classification Problems

147

On the use of ROC analysis for the optimization of abstaining classifiers

148

An Empirical Comparison of Ideal and Empirical ROC-Based Reject Rules

149

Delegating Classifiers for Automatic Text Categorization

150

Lasso type classifiers with a reject option

151

Classification with reject option

152

Bootstrap Methods for Reject Rules of Fisher LDA

153

Confidence-based classifier design

154

The interaction between classification and reject performance for distance-based reject-option classifiers

155

Optimizing abstaining classifiers using ROC analysis

156

On optimal reject rules and ROC curves

157

A ROC-based reject rule for dichotomizers

158

Improvement of reliability in banknote classification using reject option and local PCA

159

A Two-Stage Classifier with Reject Option for Text Categorisation

160

Delegating classifiers

161

Reducing the classification cost of support vector classifiers through an ROC-based reject rule

162

Analysis of error-reject trade-off in linearly combined multiple classifiers

163

Reject-Optional LVQ-Based Two-Level Classifier to Improve Reliability in Footstep Identification

164

An approach to novelty detection applied to the classification of image regions

165

Novelty detection: a review - part 1: statistical approaches

166

Classification with reject option in text categorisation systems

167

Rejection strategies and confidence measures for a k-NN classifier in an OCR task

168

Support Vector Machines with Embedded Reject Option

169

Reject Strategies Driven Combination of Pattern Classifiers

170

A Classification Reliability Driven Reject Rule for Multi-Expert Systems

171

A support vector machines-based rejection technique for speech recognition

172

Gaussian process regression: active data selection and test point rejection

173

Reject option for VQ-based Bayesian classification

174

Multiple Reject Thresholds for Improving Classification Reliability

175

An Optimal Reject Rule for Binary Classifiers

176

Efficient algorithms for mining outliers from large data sets

177

To reject or not to reject: that is the question-an answer in case of neural classifiers

178

Support Vector Method for Novelty Detection

179

Moderating the outputs of support vector machine classifiers

180

Rejection Criteria and Pairwise Discrimination of Handwritten Numerals Based on Structural Features

181

Multiclassification: reject criteria for the Bayesian combiner

182

Class-Selective Rejection Rule To Minimize The Maximum Distance Between Selected Classes

183

An Evaluation Of Multi-Expert Configurations For The Recognition Of Handwritten Numerals

184

Discriminative learning for minimum error and minimum reject classification

185

On Unifying Probabilistic/Fuzzy and Possibilistic Rejection-Based Classifiers

186

Optimizing the Error/Reject Trade-Off for a Multi-Expert System Using the Bayesian Combining Rule

187

Classifier design with incomplete knowledge

188

Optimum tradeoff between class-selective rejection error and average number of classes

189

On the Error-Reject Trade-Off in Biometric Verification Systems

190

The Optimum Class-Selective Rejection Rule

191

The Error-Reject Tradeoff

192

An optimum class-selective rejection rule for pattern recognition

193

A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise

194

An Adaptive Reject Option for LVQ Classifiers

195

A method for improving classification reliability of multilayer perceptrons

196

Optimal combinations of pattern classifiers

197

Gradient descent learning of nearest neighbor classifiers with outlier rejection

198

Investigating feedforward neural networks with respect to the rejection of spurious patterns

199

Democracy in neural nets: Voting schemes for classification

200