HiRID-ICU-Benchmark - A Comprehensive Machine Learning Benchmark on High-resolution ICU Data (2021-11-16T00:00:00.000000Z)

TL;DR

This work provides a benchmark covering a large spectrum of ICU-related tasks using the HiRID dataset, and provides an in-depth analysis of current state-of-the-art sequence modeling methods, highlighting some limitations of deep learning approaches for this type of data.

Abstract

The recent success of machine learning methods applied to time series collected from Intensive Care Units (ICU) exposes the lack of standardized machine learning benchmarks for developing and comparing such methods. While raw datasets, such as MIMIC-IV or eICU, can be freely accessed on Physionet, the choice of tasks and pre-processing is often chosen ad-hoc for each publication, limiting comparability across publications. In this work, we aim to improve this situation by providing a benchmark covering a large spectrum of ICU-related tasks. Using the HiRID dataset, we define multiple clinically relevant tasks in collaboration with clinicians. In addition, we provide a reproducible end-to-end pipeline to construct both data and labels. Finally, we provide an in-depth analysis of current state-of-the-art sequence modeling methods, highlighting some limitations of deep learning approaches for this type of data. With this benchmark, we hope to give the research community the possibility of a fair comparison of their work.

Authors

M. Faltys

2 papers

Matthias Hüser

3 papers

Xinrui Lyu

2 papers

References60 items

Clairvoyance: A Pipeline Toolkit for Medical Time Series

Neighborhood Contrastive Learning Applied to Online Patient Monitoring

Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: The Amsterdam University Medical Centers Database (AmsterdamUMCdb) Example*

TL;DR

Abstract

Authors

References60 items

Clairvoyance: A Pipeline Toolkit for Medical Time Series

Neighborhood Contrastive Learning Applied to Online Patient Monitoring

Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: The Amsterdam University Medical Centers Database (AmsterdamUMCdb) Example*

Delving into Deep Imbalanced Regression

Development and Validation of a Machine learning prediction model of respiratory failure within 48 hours of patient admission for COVID-19.

Evaluating Progress on Machine Learning for Longitudinal Electronic Healthcare Data

Systemic complement activation is associated with respiratory failure in COVID-19 hospitalized patients

Efficient Transformers: A Survey

Big Bird: Transformers for Longer Sequences

Acute respiratory failure in COVID-19: is it “typical” ARDS?

Presenting Characteristics, Comorbidities, and Outcomes Among 5700 Patients Hospitalized With COVID-19 in the New York City Area.

Efficient Content-Based Sparse Attention with Routing Transformers

Clinical predictors of mortality due to COVID-19 based on an analysis of data of 150 patients from Wuhan, China

Early prediction of circulatory failure in the intensive care unit using machine learning

The neuroinvasive potential of SARS‐CoV2 may play a role in the respiratory failure of COVID‐19 patients

5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Set Functions for Time Series

Early Prediction of Sepsis From Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019

A Clinically Applicable Approach to Continuous Prediction of Future Acute Kidney Injury

Generating Long Sequences with Sparse Transformers

Survey on deep learning with class imbalance

The eICU Collaborative Research Database, a freely available multi-center database for critical care research

Early Hospital Mortality Prediction using Vital Signals

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

LightGBM: A Highly Efficient Gradient Boosting Decision Tree

Reproducibility in critical care: a mortality prediction case study

Focal Loss for Dense Object Detection

Focal Loss for Dense Object Detection

Attention is All you Need

Multitask learning and benchmarking with clinical time series data

MIMIC-III, a freely accessible critical care database

Adam: A Method for Stochastic Optimization

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

PhysioNet 2012 Challenge: Predicting mortality of ICU patients using a cascaded SVM-GLM paradigm

Scikit-learn: Machine Learning in Python

The Balanced Accuracy and Its Posterior Distribution

Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality assessment for today’s critically ill patients*

Long Short-Term Memory

Determination of PO2 from saturation.

APACHE II: A severity of disease classification system

APACHE II: a severity of disease classification system.

HiRID, a 335 high time-resolution icu dataset (version 1.1.1)

pyts: A Python Package for Time Series Classification

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

pysf: Supervised forecasting of sequential data in python, 2018

Acute Respiratory Distress Syndrome The Berlin Definition

Logistic Regression in Rare Events Data

Physionet: components of a new research resource for complex physiologic signals

1. imputation_for_endpoints -Imputes data for endpoint generation. 2. generate_endpoints -Generates the endpoints i.e. verify if conditions for events are matched at each timestep

run_resample_step -Re-samples data to a regularly sampled 5min resolution

Remaining_LOS_Reg -Continuous prediction of the remaining stay duration (further Remaining LOS)

Gradient Boosting The range of hyperparameters considered for the gradient boosting method

generate_labels -Extract final labels from endpoints

Phenotyping_APACHEGroup -Patient APACHE group classification 24h after admission in the ICU (further Patient Phenotyping)

As mentioned in the paper, we construct a set of 6 tasks, relevant for healthcare workers: 1. Mortality_At24Hours -Mortality prediction at 24h after admission in the ICU

Hyperparameter search range for LGBM. In bold are the parameters we selected using random search. Task Depth Colsample_bytree 18

Dynamic_RespFailure_12Hour -Continuous prediction of occurrence of respiratory failure in the next 12 hours (further Respiratory Failure)

Model Training and Evaluation In the final part of the pipeline, we provide the user with a choice of model to train: Machine Learning Models Hyperparameters

Subsample 17

Field of Study

Journal Information

Name

Volume