Machine learning for medical imaging: methodological failures and recommendations for the future (2022-04-12T00:00:00.000000Z)

TL;DR

It is shown that at every step, potential biases can creep in in computer analysis of medical images, from limitations of the data to research incentives.

Abstract

Research in computer analysis of medical images bears many promises to improve patients’ health. However, a number of systematic challenges are slowing down the progress of the field, from limitations of the data, such as biases, to research incentives, such as optimizing for publication. In this paper we review roadblocks to developing and assessing methods. Building our analysis on evidence from the literature and data challenges, we show that at every step, potential biases can creep in. On a positive note, we also discuss on-going efforts to counteract these problems. Finally we provide recommendations on how to further address these problems in the future.

Authors

G. Varoquaux

3 papers

V. Cheplygina

1 papers

TL;DR

Abstract

Authors

References110 items

Preventing dataset shift from breaking machine-learning biomarkers

Minimum sample size for external validation of a clinical prediction model with a binary outcome

Accounting for Variance in Machine Learning Benchmarks

Best practices for authors of healthcare-related artificial intelligence manuscripts

I tried a bunch of things: The dangers of unexpected overfitting in classification of brain data

Evaluating Progress on Machine Learning for Longitudinal Electronic Healthcare Data

Time to reality check the promises of machine learning-powered precision medicine

Predicting the Progression of Mild Cognitive Impairment Using Machine Learning: A Systematic, Quantitative and Critical Review

Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance

Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist

Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans

A Review of Deep Learning in Medical Imaging: Imaging Traits, Technology Trends, Case Studies With Progress Highlights, and Future Promises

Threats of a replication crisis in empirical computer science

Evaluating Machine Accuracy on ImageNet

Are we done with ImageNet?

Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis

Risk of Training Diagnostic Algorithms on Data with Demographic Bias

Dataset decay and the problem of sequential analyses on open datasets

On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law

A Metric Learning Reality Check

The Problem with Metrics is a Fundamental Problem for AI

Preparing Medical Imaging Data for Machine Learning.

Detect and Correct Bias in Multi-Site Neuroimaging Datasets

Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning

A Path for Translation of Machine Learning Products into Healthcare Delivery

Sample size determination for biomedical big data with limited labels

Calibration: the Achilles heel of predictive analytics

Establishment of Best Practices for Evidence for Prediction: A Review.

Sample size evolution in neuroimaging research: An evaluation of highly-cited studies (1990–2012) and of latest practices (2017–2018) in high-impact journals

A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis.

Hidden stratification causes clinically meaningful failures in machine learning for medical imaging

Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification

Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition.

Exploring large scale public medical image datasets

Are we really making much progress? A worrying analysis of recent neural recommendation approaches

Unreproducible Research is Reproducible

The Scientific Method in the Science of Machine Learning

Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluation

HARK Side of Deep Learning - From Grad Student Descent to Automated Machine Learning

A Survey of Crowdsourcing in Medical Image Analysis

A large annotated medical image dataset for the development and evaluation of segmentation algorithms

Classification Accuracy of Neuroimaging Biomarkers in Attention-Deficit/Hyperactivity Disorder: Effects of Sample Size and Circular Analysis.

A Framework for Understanding Unintended Consequences of Machine Learning

Why Most Published Research Findings Are False

High-performance medicine: the convergence of human and artificial intelligence

PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies

Learning to Unlearn: Building Immunity to Dataset Bias in Medical Imaging Studies

Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators.

Machine learning studies on major brain diseases: 5-year trends of 2014–2018

Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift

Model Cards for Model Reporting

Inter-observer variability of manual contour delineation of structures in CT

Classify epithelium‐stroma in histopathological images based on deep transferable network

Troubling Trends in Machine Learning Scholarship

Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study

Why rankings of biomedical image analysis competitions should be interpreted with care

A Practical Taxonomy of Reproducibility for Machine Learning Research

Statistical Rituals: The Replication Delusion and How We Got There

Improving the robustness of convolutional networks to appearance variability in biomedical images

State of the Art: Reproducibility in Artificial Intelligence

Not‐so‐supervised: A survey of semi‐supervised, multi‐instance, and transfer learning in medical image analysis

Datasheets for datasets

Replication study: Development and validation of deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs

Methodologic Guide for Evaluating Clinical Performance and Effect of Artificial Intelligence Technology for Medical Diagnosis and Prediction.

How Good Is My Test Data? Introducing Safety Analysis for Computer Vision

Machine learning and microsimulation techniques on the prognosis of dementia: A systematic literature review

Cross-validation failure: Small sample sizes lead to large error bars

Confidence curves: an alternative to null hypothesis significance testing for the comparison of classifiers

Negative results in computer vision: A perspective

Ethical and Legal Implications of the Methodological Crisis in Neuroimaging

The need to approximate the use-case in clinical machine learning

Building better biomarkers: brain models in translational neuroimaging

A survey on deep learning in medical image analysis

Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls

How to Develop, Validate, and Compare Clinical Prediction Models Involving Radiological Parameters: Study Design and Statistical Methods

Hidden Technical Debt in Machine Learning Systems