Evaluation metrics and statistical tests for machine learning (2024-03-13T00:00:00.000000Z)

TL;DR

The most common evaluation metrics used for the typical supervised ML tasks including binary, multi-class, and multi-label classification, regression, image segmentation, object detection, and information retrieval are introduced.

Abstract

Research on different machine learning (ML) has become incredibly popular during the past few decades. However, for some researchers not familiar with statistics, it might be difficult to understand how to evaluate the performance of ML models and compare them with each other. Here, we introduce the most common evaluation metrics used for the typical supervised ML tasks including binary, multi-class, and multi-label classification, regression, image segmentation, object detection, and information retrieval. We explain how to choose a suitable statistical test for comparing models, how to obtain enough values of the metric for testing, and how to perform the test and interpret its results. We also present a few practical examples about comparing convolutional neural networks used to classify X-rays with different lung infections and detect cancer tumors in positron emission tomography images.

Authors

O. Rainio

author.paper_count

J. Teuho

author.paper_count

R. Klén

TL;DR

Abstract

Authors

References62 items

New method of using a convolutional neural network for 2D intraprostatic tumor segmentation from PET images

Automatic Segmentation of Head and Neck Cancer from PET-MRI Data Using Deep Learning

Classification of head and neck cancer from PET images using convolutional neural networks

Model-Based Machine Learning

Carimas: An Extensive Medical Imaging Data Processing Tool for Research

Weighted kappa measures for ordinal multi-class classification performance

Identifying Malignancy of Lung Cancer Using Deep Learning Concepts

Machine Learning for industrial applications: A comprehensive literature review

Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray images

Evaluation of machine learning algorithms for Health and Wellness applications: a tutorial

Reliable Tuberculosis Detection Using Chest X-Ray With Deep Learning, Segmentation and Visualization

A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects

Can AI Help in Screening Viral and COVID-19 Pneumonia?

Lymph node detection in MR Lymphography: false positive reduction using multi-view convolutional neural networks

SciPy 1.0: fundamental algorithms for scientific computing in Python

Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

Taming the Cross Entropy Loss

Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning

Optimal classifier for imbalanced data using Matthews Correlation Coefficient metric

Does McNemar’s test compare the sensitivities and specificities of two diagnostic tests?

Using Spearman's correlation coefficients for exploratory data analysis on big dataset

Dealing with the evaluation of supervised classification algorithms

Machine learning: Trends, perspectives, and prospects

U-Net: Convolutional Networks for Biomedical Image Segmentation

Statistical and Machine Learning Approaches for Network Analysis

On the Mathematical Properties of the Structural Similarity Index

pROC: an open-source package for R and S+ to analyze and compare ROC curves

Nonparametric Statistics for Non-Statisticians: A Step-by-Step Approach

Python 3 Reference Manual

Estimation and Comparison of Receiver Operating Characteristic Curves

Measures of Diagnostic Accuracy: Basic Definitions

Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test

McNemar χ2 test revisited: comparing sensitivity and specificity of diagnostic examinations

Statistical Comparisons of Classifiers over Multiple Data Sets

Statistical precision of information retrieval evaluation

Optimal feature combination for automated segmentation of prostatic adenocarcinoma from high resolution MRI

Foundations of Statistical Natural Language Processing

On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach

Epidemiology, Biostatistics and Preventive Medicine

Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.

Statistical Methods for Engineers and Scientists.

An Analysis of Variance Test for Normality (Complete Samples)

A Coefficient of Agreement for Nominal Scales

Measures of the Amount of Ecologic Association Between Species

THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE.1

Elements of Data Science, Machine Learning, and Artificial Intelligence Using R

Machine Learning with R: Learn Techniques for Building and Improving Machine Learning Models,

Laboratory Screening and Diagnostic Evaluation: An Evidence-Based Approach

R Core Team

Early History of Machine Learning

Hands-On Computer Vision with TensorFlow 2: Leverage Deep Learning to Create Powerful Image Processing Apps with TensorFlow 2.0 and Keras

TensorFlow: Large-scale machine learning on heterogeneous systems

An Overview of Current Evaluation Methods Used in Medical Image Segmentation

Model Based Comparison of Discounted Cumulative Gain and Average Precision

Statsmodels: Econometric and Statistical Modeling with Python

Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS implementations

How to Report Statistics in Medicine

Properties of Sufficiency and Statistical Tests

Approximations of the critical region of the fbietkan statistic

Robust tests for equality of variances

A method of establishing group of equal amplitude in plant sociobiology based on similarity of species content and its application to analyses of the vegetation on Danish commons

Intelligent image segmentation methods using deep convolutional neural network

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names