This survey aims to provide an overview on machine learning with rejection, and introduces the conditions leading to two types of rejection, ambiguity and novelty rejection, which are carefully formalized.
Asymmetric Distribution Measure for Few-shot Learning
51
Arbitrated Dynamic Ensemble with Abstaining for Time-Series Forecasting on Data Streams
52
Performance visualization spaces for classification with rejection option
53
Active Learning with Abstaining Classifiers for Imbalanced Drifting Data Streams
54
Aleatoric and Epistemic Uncertainty in Machine Learning: A Tutorial Introduction
55
Deep neural rejection against adversarial examples
56
Epistemic Uncertainty Sampling
57
Deep Gamblers: Learning to Abstain with Portfolio Theory
58
Combating Label Noise in Deep Learning Using Abstention
59
On discriminative learning of prediction uncertainty
60
Binary Classification with Bounded Abstention Rate
61
Reliable Multi-label Classification: Prediction with Partial Abstention
62
Interpretable Cascade Classifiers with Abstention
63
Hybrid Models with Deep and Invertible Features
64
On the Calibration of Multiclass Classification with Rejection
65
SelectiveNet: A Deep Neural Network with an Integrated Reject Option
66
Semi-Supervised Anomaly Detection with an Application to Water Analytics
67
Meta-Learning: A Survey
68
Mitigating Concept Drift via Rejection
69
Rotation-blended CNNs on a New Open Dataset for Tropical Cyclone Image-to-intensity Regression
70
Uncertainty-Aware Attention for Reliable Interpretation and Prediction
71
COBRAS-TS: A new approach to Semi-Supervised Clustering of Time Series
72
To Trust Or Not To Trust A Classifier
73
Interpretable machine learning with reject option
74
Online ensemble learning with abstaining classifiers for drifting and noisy data streams
75
Machine Learning with Abstention for Automated Liver Disease Diagnosis
76
Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer
77
SafePredict: A Meta-Algorithm for Machine Learning That Uses Refusals to Guarantee Correctness
78
Probabilistic extension and reject options for pairwise LVQ
79
Concrete Dropout
80
Twin SVM with a reject option through ROC curve
81
Selective Classification for Deep Neural Networks
82
Blocking Transferability of Adversarial Examples in Black-Box Learning Systems
83
Online Learning with Abstention
84
On Reject and Refine Options in Multicategory Classification
85
Classification with reject option for software defect prediction
86
ROC-based cost-sensitive classification with a reject option
87
Background Check: A General Technique to Build More Reliable and Versatile Classifiers
88
Optimal local rejection for classifiers
89
Reliable prediction of anti-diabetic drug failure using a reject option
90
Learning with Rejection
91
Local Reject Option for Deterministic Multi-class SVM
92
Evidence-based uncertainty sampling for active learning
93
Learning to Abstain from Binary Prediction
94
Efficient rejection strategies for prototype-based classification
95
Datasets meta-feature description for recommending feature selection algorithm
96
Robust classification with reject option using the self-organizing map
97
Classification with a reject option under Concept Drift: The Droplets algorithm
98
Image Classification with Rejection using Contextual Information
99
Learning Vector Quantization with Adaptive Cost-Based Outlier-Rejection
100
Consistency of plug-in confidence sets for classification in semi-supervised learning
101
Supervised hyperspectral image classification with rejection
102
Siamese neural network based similarity metric for inertial gesture classification and rejection
103
Robust hyperspectral image classification with rejection fields
104
Performance measures for classification systems with rejection
105
Optimum Reject Options for Prototype-based Classification
106
Classification with confidence
107
Classification with Reject Option Using the Self-Organizing Map
108
Local Rejection Strategies for Learning Vector Quantization
109
Combination of One-Class Support Vector Machines for Classification with Reject Option
110
Random Forest for Reliable Pre-classification of Handwritten Characters
111
Beyond Disagreement-Based Agnostic Active Learning
112
Classification with rejection based on various SVM techniques
113
A review of novelty detection
114
Highly Accurate Recognition of Human Postures and Activities Through Classification With Rejection
115
Rejection Schemes in Multi-class Classification -- Application to Handwritten Character Recognition
116
Multi-label classification with a reject option
117
Classifiers with a reject option for early time-series classification
118
Classification with reject option using contextual information
119
Reliable Classification of Vehicle Types Based on Cascade Classifier Ensembles
120
Weighted extreme learning machine for imbalance learning
121
MetaFraud: A Meta-Learning Framework for Detecting Financial Fraud
122
Multiple classifier combination using reject options and markov fusion networks
123
An Ensemble of Rejecting Classifiers for Anomaly Detection of Audio Events
124
Heartbeat Classification Using Support Vector Machines (SVMs) with an Embedded Reject Option
125
Design of reject rules for ECOC classification systems
126
Agnostic Selective Classification
127
A Classification Approach with a Reject Option for Multi-label Problems
128
A risk bound for ensemble classification with a reject option
129
Diagnostic of Pathology on the Vertebral Column with Embedded Reject Option
130
A Rejection Option for the Multilayer Perceptron Using Hyperplanes
131
Trading off Mistakes and Don't-Know Predictions
132
An Introduction to Conditional Random Fields
133
The data replication method for the classification with reject option
134
Adapting cost-sensitive learning for reject option
135
An Optimum Class-Rejective Decision Rule and Its Evaluation
136
On the Foundations of Noise-free Selective Classification
137
Classification Methods with Reject Option Based on Convex Risk Minimization
138
An Ordinal Data Method for the Classification with Reject Option
139
Accuracy-Rejection Curves (ARCs) for Comparing Classification Methods with a Reject Option
140
Isolation Forest
141
Pattern rejection strategies for the design of self-paced EEG-based Brain-Computer Interfaces
142
Support Vector Machines with a Reject Option
143
Multi-level Classification of Emphysema in HRCT Lung Images Using Delegated Classifiers
144
Classification with reject option in gene expression data
145
General solution and learning method for binary classification with performance constraints
146
Growing a multi-class classifier with a reject option
147
Classification with a Reject Option using a Hinge Loss
148
A Kernel Based Rejection Method for Supervised Classification
149
Learning to Classify Ordinal Data: The Data Replication Method
150
A Novel Classification-Rejection Sphere SVMs for Multi-class Classification Problems
151
On the use of ROC analysis for the optimization of abstaining classifiers
152
An Empirical Comparison of Ideal and Empirical ROC-Based Reject Rules
153
Delegating Classifiers for Automatic Text Categorization
154
Lasso type classifiers with a reject option
155
Classification with reject option
156
Bootstrap Methods for Reject Rules of Fisher LDA
157
Confidence-based classifier design
158
The interaction between classification and reject performance for distance-based reject-option classifiers
159
Optimizing abstaining classifiers using ROC analysis
160
On optimal reject rules and ROC curves
161
A ROC-based reject rule for dichotomizers
162
Improvement of reliability in banknote classification using reject option and local PCA
163
A Two-Stage Classifier with Reject Option for Text Categorisation
164
Delegating classifiers
165
Reducing the classification cost of support vector classifiers through an ROC-based reject rule
166
Analysis of error-reject trade-off in linearly combined multiple classifiers
167
Reject-Optional LVQ-Based Two-Level Classifier to Improve Reliability in Footstep Identification
168
An approach to novelty detection applied to the classification of image regions
169
Novelty detection: a review - part 1: statistical approaches
170
Classification with reject option in text categorisation systems
171
Rejection strategies and confidence measures for a k-NN classifier in an OCR task
172
Support Vector Machines with Embedded Reject Option
173
Reject Strategies Driven Combination of Pattern Classifiers
174
A Classification Reliability Driven Reject Rule for Multi-Expert Systems
175
A support vector machines-based rejection technique for speech recognition
176
Gaussian process regression: active data selection and test point rejection
177
Reject option for VQ-based Bayesian classification
178
Multiple Reject Thresholds for Improving Classification Reliability
179
An Optimal Reject Rule for Binary Classifiers
180
Efficient algorithms for mining outliers from large data sets
181
To reject or not to reject: that is the question-an answer in case of neural classifiers
182
Support Vector Method for Novelty Detection
183
Moderating the outputs of support vector machine classifiers
184
Rejection Criteria and Pairwise Discrimination of Handwritten Numerals Based on Structural Features
185
Multiclassification: reject criteria for the Bayesian combiner
186
Class-Selective Rejection Rule To Minimize The Maximum Distance Between Selected Classes
187
An Evaluation Of Multi-Expert Configurations For The Recognition Of Handwritten Numerals
188
Discriminative learning for minimum error and minimum reject classification
189
On Unifying Probabilistic/Fuzzy and Possibilistic Rejection-Based Classifiers
190
Optimizing the Error/Reject Trade-Off for a Multi-Expert System Using the Bayesian Combining Rule
191
Classifier design with incomplete knowledge
192
Optimum tradeoff between class-selective rejection error and average number of classes
193
On the Error-Reject Trade-Off in Biometric Verification Systems
194
The Optimum Class-Selective Rejection Rule
195
The Error-Reject Tradeoff
196
An optimum class-selective rejection rule for pattern recognition
197
An Adaptive Reject Option for LVQ Classifiers
198
A method for improving classification reliability of multilayer perceptrons
199
Optimal combinations of pattern classifiers
200
Gradient descent learning of nearest neighbor classifiers with outlier rejection
201
Investigating feedforward neural networks with respect to the rejection of spurious patterns
202
Democracy in neural nets: Voting schemes for classification
203
Enhanced reliability of multilayer perceptron networks through controlled pattern rejection
204
Multistage pattern recognition with reject option
205
Methods of combining multiple classifiers and their applications to handwriting recognition
206
Handwritten zip code recognition with multilayer networks
207
A perspective on artificial intelligence: Learning to learn
208
Application of optimum error-reject functions (Corresp.)
209
The Nearest Neighbor Classification Rule with a Reject Option
210
An optimum character recognition system using decision functions
211
Detecting Evasion Attacks in Deployed Tree Ensembles
212
Semi-supervised Learning from Active Noisy Soft Labels for Anomaly Detection
213
A novel reject option applied to sleep stage scoring
214
Reject Before You Run: Small Assessors Anticipate Big Language Models
215
Generalizing Consistent Multi-Class Classification with Rejection to be Compatible with Arbitrary Losses
216
Combining Base-Learners into Ensembles
217
Adversarial Training with Rectified Rejection
218
Learning with Labeling Induced Abstentions
219
Know Your Limits: Machine Learning with Rejection for Vehicle Engineering
220
Fair Selective Classification Via Sufficiency
221
Towards optimally abstaining from prediction with OOD test examples
222
A Ranking Stability Measure for Quantifying the Robustness of Anomaly Detection Methods
223
Quantifying the Confidence of Anomaly Detectors in Their Example-Wise Predictions
224
Accuracy Rejection Normalized-Cost Curves (ARNCCs): A Novel 3-Dimensional Framework for Robust Classification
225
Consistent algorithms for multiclass classification with an abstain option
226
Supplementary Materials of “On Reject and Refine Options in Multicategory Classification”
227
Hyperparameter Optimization
228
Boosting with Abstention
229
Self-Adjusting Reject Options in Prototype Based Classification
230
A Probabilistic Classifier Model with Adaptive Rejection Option Report 01 / 2016
231
Reject Option Paradigm for the Reduction of Support Vectors
232
Rejection Strategies for Learning Vector Quantization - A Comparison of Probabilistic and Deterministic Approaches
233
Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty
234
Classification with Rejection: Concepts and Evaluations
235
A survey on instance selection for active learning
236
A family of measures for best top-n class-selective decision rules
237
Cost-sensitive Multi-class SVM with Reject Option: A Method for Steam Turbine Generator Fault Diagnosis
238
Cost-sensitive SVM with Error Cost and Class-dependent Reject Cost
239
Reject Options and Confidence Measures for kNN Classifiers
240
Linear Classifier with Reject Option for the Detection of Vocal Fold Paralysis and Vocal Fold Edema
241
Metalearning - Applications to Data Mining
242
Sleep Versus Wake Classification From Heart Rate Variability Using Computational Intelligence: Consideration of Rejection in Classification Models
243
Aggregating Abstaining and Delegating Classifiers For Improving Classification performance : An application to lung cancer survival prediction
244
An Introduction to Conditional Random Fields for Relational Learning
245
An Introduction to Conditional Random Fields for Relational Learning
246
Cost Curves for Abstaining Classifiers
247
RO-SVM: Support Vector Machine with Reject Option for Image Categorization
248
A combining strategy for ill-defined problems
249
An Analysis of Reliable Classifiers through ROC Isometrics
250
Reliable Classifiers in ROC Space
251
Cautious Classifiers
252
Neural Computing: New Challenges and Perspectives for the New Millennium
253
LOF: Identifying DensityBased Local Outliers
254
Probabilistic measures for responses of Self-Organizing Map units
255
Learning rejection thresholds for a class of fuzzy classifiers from possibilistic clustered noisy data
256
A Method of Combining Multiple Experts for the Recognition of Unconstrained Handwritten Numerals
257
Trading off coverage for accuracy in forecasts: Applications to clinical data analysis
258
A statistical decision rule with incomplete knowledge about classes
259
Computer aided system diagnostic with an incomplete learning set
260
BAYESIAN DECISION WITH REJECTION.
261
On optimum recognition error and reject tradeoff
262
Aggregating Abstaining and Delegating Classifiers for Improving Classification Performance: an Application to Lung Cancer Survival Prediction
263
Ambiguity Rejection: if Y | x has a high variance or h ( x ) has a high bias, which happens when x falls in a region where the target value is ambiguous
264
Q1. How can we formalize the conditions for which a model should abstain from making a prediction?
265
Q6. How can we combine multiple rejectors?
266
Q2. How can we evaluate the performance of a model with rejection?
267
Q7. Where does the need for machine learning with rejection methods arise in real-world applications?
268
Q3. What architectures are possible for operationalizing (i.e., putting this into practice) the ability to abstain from making a prediction?
269
The IEEE International Conference on Data Science and Advanced Analytics
270
Ambiguity Rejection: occurs if x falls in a region where the target y is ambiguous ( R1 and R2 )
271
R4. Some examples x could simply not be acquired due to their inherent rarity (anomalies, out-of-distribution)
272
Based on this intuition, two types of rejections can be performed
273
R2. Some instances in the training data are incorrect
274
R1. There can be cases where a vector x i ∈ X is associated with multiple values from the target space Y . This can arise in situations such as when classes overlap in classification tasks
275
Q5. What are the main pros and cons of using a specific architecture?