Research Connect
Research PapersAboutContact

Machine Learning with a Reject Option: A survey

Published in Machine-mediated learning (2021-07-23)
aionlincourseaionlincourseaionlincourseaionlincourseaionlincourseaionlincourse
Generate GraphDownload

On This Page

  • TL;DR
  • Abstract
  • Authors
  • Datasets
  • References
TL

TL;DR

This survey aims to provide an overview on machine learning with rejection, and introduces the conditions leading to two types of rejection, ambiguity and novelty rejection, which are carefully formalized.

Abstract

Authors

Kilian Hendrickx

1 Paper

Lorenzo Perini

1 Paper

Dries Van der Plas

1 Paper

Wannes Meert

1 Paper

Jesse Davis

1 Paper

References276 items

1

A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise

2

A comprehensive survey and analysis of generative models in machine learning

3

Active Learning Literature Survey

4

Neural Networks and the Bias/Variance Dilemma

5

Can We Trust Fair-AI?

6

Research Impact

102

Citations

276

References

0

Datasets

5

A Model-Agnostic Heuristics for Selective Classification

7

Sources of Uncertainty in Machine Learning - A Statisticians' View

8

How to Allocate your Label Budget? Choosing between Active Learning and Learning to Reject in Anomaly Detection

9

Estimating the Contamination Factor's Distribution in Unsupervised Anomaly Detection

10

AUC-based Selective Classification

11

Transferring the Contamination Factor between Anomaly Detection Domains by Shape Similarity

12

A meta-learning BCI for estimating decision confidence

13

Towards Better Selective Classification

14

Nonparametric Uncertainty Quantification for Single Deterministic Neural Network

15

Uncertainty-Based Rejection in Machine Learning: Implications for Model Development and Interpretability

16

Management of validation of HPLC method for determination of acetylsalicylic acid impurities in a new pharmaceutical product

17

Multilabel Classification with Partial Abstention: Bayes-Optimal Prediction under Label Independence

18

Online Selective Classification with Limited Feedback

19

Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis

20

Bayesian Active Meta-Learning for Reliable and Efficient AI-Based Demodulation

21

Securing Deep Learning Models with Autoencoder based Anomaly Detection

22

Two Coupled Rejection Metrics Can Tell Adversarial Examples Apart

23

Probabilistic personalised cascade with abstention

24

SCRIB: Set-classifier with Class-specific Risk Bounds for Blackbox Models

25

Exponential Savings in Agnostic Active Learning Through Abstention

26

Optimal strategies for reject option classifiers

27

How to define a rejection class based on model learning?

28

Second opinion needed: communicating uncertainty in medical machine learning

29

Fast and Accurate $k$-means++ via Rejection Sampling

30

Anomaly Detection

31

Confidence Estimation via Auxiliary Models

32

Know your limits: Uncertainty estimation with ReLU classifiers fails at reliable OOD detection

33

Versatile Verification of Tree Ensembles

34

ATRO: Adversarial Training with a Rejection Option

35

Classification with Rejection Based on Cost-sensitive Classification

36

Selective Classification via One-Sided Prediction

37

Probability of default estimation, with a reject option

38

Generalized Neural Framework for Learning with Rejection

39

Classification with Rejection: Scaling Generative Classifiers with Supervised Deep Infomax

40

Risk-Controlled Selective Prediction for Regression Deep Neural Network Models

41

Classification Under Human Assistance

42

Consistent Estimators for Learning to Defer to an Expert

43

Bounded–abstaining classification for breast tumors in imbalanced ultrasound images

44

Active Learning for Classification with Abstention

45

Uncertainty-Based Rejection Wrappers for Black-Box Classifiers

46

A Novel Meta Learning Framework for Feature Selection using Data Synthesis and Fuzzy Similarity

47

AUC Optimization with a Reject Option

48

Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data

49

Self-Adaptive Training: beyond Empirical Risk Minimization

50

Asymmetric Distribution Measure for Few-shot Learning

51

Arbitrated Dynamic Ensemble with Abstaining for Time-Series Forecasting on Data Streams

52

Performance visualization spaces for classification with rejection option

53

Active Learning with Abstaining Classifiers for Imbalanced Drifting Data Streams

54

Aleatoric and Epistemic Uncertainty in Machine Learning: A Tutorial Introduction

55

Deep neural rejection against adversarial examples

56

Epistemic Uncertainty Sampling

57

Deep Gamblers: Learning to Abstain with Portfolio Theory

58

Combating Label Noise in Deep Learning Using Abstention

59

On discriminative learning of prediction uncertainty

60

Binary Classification with Bounded Abstention Rate

61

Reliable Multi-label Classification: Prediction with Partial Abstention

62

Interpretable Cascade Classifiers with Abstention

63

Hybrid Models with Deep and Invertible Features

64

On the Calibration of Multiclass Classification with Rejection

65

SelectiveNet: A Deep Neural Network with an Integrated Reject Option

66

Semi-Supervised Anomaly Detection with an Application to Water Analytics

67

Meta-Learning: A Survey

68

Mitigating Concept Drift via Rejection

69

Rotation-blended CNNs on a New Open Dataset for Tropical Cyclone Image-to-intensity Regression

70

Uncertainty-Aware Attention for Reliable Interpretation and Prediction

71

COBRAS-TS: A new approach to Semi-Supervised Clustering of Time Series

72

To Trust Or Not To Trust A Classifier

73

Interpretable machine learning with reject option

74

Online ensemble learning with abstaining classifiers for drifting and noisy data streams

75

Machine Learning with Abstention for Automated Liver Disease Diagnosis

76

Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer

77

SafePredict: A Meta-Algorithm for Machine Learning That Uses Refusals to Guarantee Correctness

78

Probabilistic extension and reject options for pairwise LVQ

79

Concrete Dropout

80

Twin SVM with a reject option through ROC curve

81

Selective Classification for Deep Neural Networks

82

Blocking Transferability of Adversarial Examples in Black-Box Learning Systems

83

Online Learning with Abstention

84

On Reject and Refine Options in Multicategory Classification

85

Classification with reject option for software defect prediction

86

ROC-based cost-sensitive classification with a reject option

87

Background Check: A General Technique to Build More Reliable and Versatile Classifiers

88

Optimal local rejection for classifiers

89

Reliable prediction of anti-diabetic drug failure using a reject option

90

Learning with Rejection

91

Local Reject Option for Deterministic Multi-class SVM

92

Evidence-based uncertainty sampling for active learning

93

Learning to Abstain from Binary Prediction

94

Efficient rejection strategies for prototype-based classification

95

Datasets meta-feature description for recommending feature selection algorithm

96

Robust classification with reject option using the self-organizing map

97

Classification with a reject option under Concept Drift: The Droplets algorithm

98

Image Classification with Rejection using Contextual Information

99

Learning Vector Quantization with Adaptive Cost-Based Outlier-Rejection

100

Consistency of plug-in confidence sets for classification in semi-supervised learning

101

Supervised hyperspectral image classification with rejection

102

Siamese neural network based similarity metric for inertial gesture classification and rejection

103

Robust hyperspectral image classification with rejection fields

104

Performance measures for classification systems with rejection

105

Optimum Reject Options for Prototype-based Classification

106

Classification with confidence

107

Classification with Reject Option Using the Self-Organizing Map

108

Local Rejection Strategies for Learning Vector Quantization

109

Combination of One-Class Support Vector Machines for Classification with Reject Option

110

Random Forest for Reliable Pre-classification of Handwritten Characters

111

Beyond Disagreement-Based Agnostic Active Learning

112

Classification with rejection based on various SVM techniques

113

A review of novelty detection

114

Highly Accurate Recognition of Human Postures and Activities Through Classification With Rejection

115

Rejection Schemes in Multi-class Classification -- Application to Handwritten Character Recognition

116

Multi-label classification with a reject option

117

Classifiers with a reject option for early time-series classification

118

Classification with reject option using contextual information

119

Reliable Classification of Vehicle Types Based on Cascade Classifier Ensembles

120

Weighted extreme learning machine for imbalance learning

121

MetaFraud: A Meta-Learning Framework for Detecting Financial Fraud

122

Multiple classifier combination using reject options and markov fusion networks

123

An Ensemble of Rejecting Classifiers for Anomaly Detection of Audio Events

124

Heartbeat Classification Using Support Vector Machines (SVMs) with an Embedded Reject Option

125

Design of reject rules for ECOC classification systems

126

Agnostic Selective Classification

127

A Classification Approach with a Reject Option for Multi-label Problems

128

A risk bound for ensemble classification with a reject option

129

Diagnostic of Pathology on the Vertebral Column with Embedded Reject Option

130

A Rejection Option for the Multilayer Perceptron Using Hyperplanes

131

Trading off Mistakes and Don't-Know Predictions

132

An Introduction to Conditional Random Fields

133

The data replication method for the classification with reject option

134

Adapting cost-sensitive learning for reject option

135

An Optimum Class-Rejective Decision Rule and Its Evaluation

136

On the Foundations of Noise-free Selective Classification

137

Classification Methods with Reject Option Based on Convex Risk Minimization

138

An Ordinal Data Method for the Classification with Reject Option

139

Accuracy-Rejection Curves (ARCs) for Comparing Classification Methods with a Reject Option

140

Isolation Forest

141

Pattern rejection strategies for the design of self-paced EEG-based Brain-Computer Interfaces

142

Support Vector Machines with a Reject Option

143

Multi-level Classification of Emphysema in HRCT Lung Images Using Delegated Classifiers

144

Classification with reject option in gene expression data

145

General solution and learning method for binary classification with performance constraints

146

Growing a multi-class classifier with a reject option

147

Classification with a Reject Option using a Hinge Loss

148

A Kernel Based Rejection Method for Supervised Classification

149

Learning to Classify Ordinal Data: The Data Replication Method

150

A Novel Classification-Rejection Sphere SVMs for Multi-class Classification Problems

151

On the use of ROC analysis for the optimization of abstaining classifiers

152

An Empirical Comparison of Ideal and Empirical ROC-Based Reject Rules

153

Delegating Classifiers for Automatic Text Categorization

154

Lasso type classifiers with a reject option

155

Classification with reject option

156

Bootstrap Methods for Reject Rules of Fisher LDA

157

Confidence-based classifier design

158

The interaction between classification and reject performance for distance-based reject-option classifiers

159

Optimizing abstaining classifiers using ROC analysis

160

On optimal reject rules and ROC curves

161

A ROC-based reject rule for dichotomizers

162

Improvement of reliability in banknote classification using reject option and local PCA

163

A Two-Stage Classifier with Reject Option for Text Categorisation

164

Delegating classifiers

165

Reducing the classification cost of support vector classifiers through an ROC-based reject rule

166

Analysis of error-reject trade-off in linearly combined multiple classifiers

167

Reject-Optional LVQ-Based Two-Level Classifier to Improve Reliability in Footstep Identification

168

An approach to novelty detection applied to the classification of image regions

169

Novelty detection: a review - part 1: statistical approaches

170

Classification with reject option in text categorisation systems

171

Rejection strategies and confidence measures for a k-NN classifier in an OCR task

172

Support Vector Machines with Embedded Reject Option

173

Reject Strategies Driven Combination of Pattern Classifiers

174

A Classification Reliability Driven Reject Rule for Multi-Expert Systems

175

A support vector machines-based rejection technique for speech recognition

176

Gaussian process regression: active data selection and test point rejection

177

Reject option for VQ-based Bayesian classification

178

Multiple Reject Thresholds for Improving Classification Reliability

179

An Optimal Reject Rule for Binary Classifiers

180

Efficient algorithms for mining outliers from large data sets

181

To reject or not to reject: that is the question-an answer in case of neural classifiers

182

Support Vector Method for Novelty Detection

183

Moderating the outputs of support vector machine classifiers

184

Rejection Criteria and Pairwise Discrimination of Handwritten Numerals Based on Structural Features

185

Multiclassification: reject criteria for the Bayesian combiner

186

Class-Selective Rejection Rule To Minimize The Maximum Distance Between Selected Classes

187

An Evaluation Of Multi-Expert Configurations For The Recognition Of Handwritten Numerals

188

Discriminative learning for minimum error and minimum reject classification

189

On Unifying Probabilistic/Fuzzy and Possibilistic Rejection-Based Classifiers

190

Optimizing the Error/Reject Trade-Off for a Multi-Expert System Using the Bayesian Combining Rule

191

Classifier design with incomplete knowledge

192

Optimum tradeoff between class-selective rejection error and average number of classes

193

On the Error-Reject Trade-Off in Biometric Verification Systems

194

The Optimum Class-Selective Rejection Rule

195

The Error-Reject Tradeoff

196

An optimum class-selective rejection rule for pattern recognition

197

An Adaptive Reject Option for LVQ Classifiers

198

A method for improving classification reliability of multilayer perceptrons

199

Optimal combinations of pattern classifiers

200

Gradient descent learning of nearest neighbor classifiers with outlier rejection

201

Investigating feedforward neural networks with respect to the rejection of spurious patterns

202

Democracy in neural nets: Voting schemes for classification

203

Enhanced reliability of multilayer perceptron networks through controlled pattern rejection

204

Multistage pattern recognition with reject option

205

Methods of combining multiple classifiers and their applications to handwriting recognition

206

Handwritten zip code recognition with multilayer networks

207

A perspective on artificial intelligence: Learning to learn

208

Application of optimum error-reject functions (Corresp.)

209

The Nearest Neighbor Classification Rule with a Reject Option

210

An optimum character recognition system using decision functions

211

Detecting Evasion Attacks in Deployed Tree Ensembles

212

Semi-supervised Learning from Active Noisy Soft Labels for Anomaly Detection

213

A novel reject option applied to sleep stage scoring

214

Reject Before You Run: Small Assessors Anticipate Big Language Models

215

Generalizing Consistent Multi-Class Classification with Rejection to be Compatible with Arbitrary Losses

216

Combining Base-Learners into Ensembles

217

Adversarial Training with Rectified Rejection

218

Learning with Labeling Induced Abstentions

219

Know Your Limits: Machine Learning with Rejection for Vehicle Engineering

220

Fair Selective Classification Via Sufficiency

221

Towards optimally abstaining from prediction with OOD test examples

222

A Ranking Stability Measure for Quantifying the Robustness of Anomaly Detection Methods

223

Quantifying the Confidence of Anomaly Detectors in Their Example-Wise Predictions

224

Accuracy Rejection Normalized-Cost Curves (ARNCCs): A Novel 3-Dimensional Framework for Robust Classification

225

Consistent algorithms for multiclass classification with an abstain option

226

Supplementary Materials of “On Reject and Refine Options in Multicategory Classification”

227

Hyperparameter Optimization

228

Boosting with Abstention

229

Self-Adjusting Reject Options in Prototype Based Classification

230

A Probabilistic Classifier Model with Adaptive Rejection Option Report 01 / 2016

231

Reject Option Paradigm for the Reduction of Support Vectors

232

Rejection Strategies for Learning Vector Quantization - A Comparison of Probabilistic and Deterministic Approaches

233

Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty

234

Classification with Rejection: Concepts and Evaluations

235

A survey on instance selection for active learning

236

A family of measures for best top-n class-selective decision rules

237

Cost-sensitive Multi-class SVM with Reject Option: A Method for Steam Turbine Generator Fault Diagnosis

238

Cost-sensitive SVM with Error Cost and Class-dependent Reject Cost

239

Reject Options and Confidence Measures for kNN Classifiers

240

Linear Classifier with Reject Option for the Detection of Vocal Fold Paralysis and Vocal Fold Edema

241

Metalearning - Applications to Data Mining

242

Sleep Versus Wake Classification From Heart Rate Variability Using Computational Intelligence: Consideration of Rejection in Classification Models

243

Aggregating Abstaining and Delegating Classifiers For Improving Classification performance : An application to lung cancer survival prediction

244

An Introduction to Conditional Random Fields for Relational Learning

245

An Introduction to Conditional Random Fields for Relational Learning

246

Cost Curves for Abstaining Classifiers

247

RO-SVM: Support Vector Machine with Reject Option for Image Categorization

248

A combining strategy for ill-defined problems

249

An Analysis of Reliable Classifiers through ROC Isometrics

250

Reliable Classifiers in ROC Space

251

Cautious Classifiers

252

Neural Computing: New Challenges and Perspectives for the New Millennium

253

LOF: Identifying DensityBased Local Outliers

254

Probabilistic measures for responses of Self-Organizing Map units

255

Learning rejection thresholds for a class of fuzzy classifiers from possibilistic clustered noisy data

256

A Method of Combining Multiple Experts for the Recognition of Unconstrained Handwritten Numerals

257

Trading off coverage for accuracy in forecasts: Applications to clinical data analysis

258

A statistical decision rule with incomplete knowledge about classes

259

Computer aided system diagnostic with an incomplete learning set

260

BAYESIAN DECISION WITH REJECTION.

261

On optimum recognition error and reject tradeoff

262

Aggregating Abstaining and Delegating Classifiers for Improving Classification Performance: an Application to Lung Cancer Survival Prediction

263

Ambiguity Rejection: if Y | x has a high variance or h ( x ) has a high bias, which happens when x falls in a region where the target value is ambiguous

264

Q1. How can we formalize the conditions for which a model should abstain from making a prediction?

265

Q6. How can we combine multiple rejectors?

266

Q2. How can we evaluate the performance of a model with rejection?

267

Q7. Where does the need for machine learning with rejection methods arise in real-world applications?

268

Q3. What architectures are possible for operationalizing (i.e., putting this into practice) the ability to abstain from making a prediction?

269

The IEEE International Conference on Data Science and Advanced Analytics

270

Ambiguity Rejection: occurs if x falls in a region where the target y is ambiguous ( R1 and R2 )

271

R4. Some examples x could simply not be acquired due to their inherent rarity (anomalies, out-of-distribution)

272

Based on this intuition, two types of rejections can be performed

273

R2. Some instances in the training data are incorrect

274

R1. There can be cases where a vector x i ∈ X is associated with multiple values from the target space Y . This can arise in situations such as when classes overlap in classification tasks

275

Q5. What are the main pros and cons of using a specific architecture?

276

Separated rejector architecture

Authors

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2107.11277

Venue Information

Name

Machine-mediated learning

Type

journal

URL

http://www.springer.com/computer/artificial/journal/10994

Alternate Names

  • Mach learn
  • Machine Learning
  • Mach Learn