Relaxed Parameter Sharing: Effectively Modeling Time-Varying Relationships in Clinical Time-Series (2019-01-01T00:00:00.000000Z)

TL;DR

A novel RNN formulation based on a mixture model in which relaxed parameter sharing over time is proposed, which outperforms standard LSTMs and other state-of-the-art baselines across all tasks and can lead to improved patient risk stratification performance.

Abstract

Recurrent neural networks (RNNs) are commonly applied to clinical time-series data with the goal of learning patient risk stratification models. Their effectiveness is due, in part, to their use of parameter sharing over time (i.e., cells are repeated hence the name recurrent). We hypothesize, however, that this trait also contributes to the increased difficulty such models have with learning relationships that change over time. Conditional shift, i.e., changes in the relationship between the input X and the output y, arises when risk factors associated with the event of interest change over the course of a patient admission. While in theory, RNNs and gated RNNs (e.g., LSTMs) in particular should be capable of learning time-varying relationships, when training data are limited, such models often fail to accurately capture these dynamics. We illustrate the advantages and disadvantages of complete parameter sharing (RNNs) by comparing an LSTM with shared parameters to a sequential architecture with time-varying parameters on prediction tasks involving three clinically-relevant outcomes: acute respiratory failure (ARF), shock, and in-hospital mortality. In experiments using synthetic data, we demonstrate how parameter sharing in LSTMs leads to worse performance in the presence of conditional shift. To improve upon the dichotomy between complete parameter sharing and no parameter sharing, we propose a novel RNN formulation based on a mixture model in which we relax parameter sharing over time. The proposed method outperforms standard LSTMs and other state-of-the-art baselines across all tasks. In settings with limited data, relaxed parameter sharing can lead to improved patient risk stratification performance.

Authors

References45 items

Learning Implicitly Recurrent CNNs Through Parameter Sharing

Can Deep Clinical Models Handle Real-World Domain Shifts?

Understanding Behavior of Clinical Models under Domain Shifts

Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts

Adapting to Concept Drift in Credit Card Transaction Data Streams Using Contextual Bandits and Decision Trees

Scalable and accurate deep learning with electronic health records

All Models are Wrong but many are Useful: Variable Importance for Black-Box, Proprietary, or Misspecified Prediction Models, using Model Class Reliance

Model Class Reliance: Variable Importance Measures for any Machine Learning Model Class, from the "Rashomon" Perspective

SkipNet: Learning Dynamic Routing in Convolutional Networks

Interpretation of Neural Networks is Fragile

Early Improving Recurrent Elastic Highway Network

Attention is All you Need

ShortFuse: Biomedical Time Series Representations in the Presence of Structured Information

Multitask learning and benchmarking with clinical time series data

Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations

Recurrent Neural Networks with Auxiliary Labels for Cross-Domain Opinion Target Extraction

Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization

HyperNetworks

Fast Unsupervised Online Drift Detection Using Incremental Kolmogorov-Smirnov Test

Domain Adaptation with Conditional Transferable Components

MIMIC-III, a freely accessible critical care database

Cluster Adaptive Training for Deep Neural Network Based Acoustic Model

Unitary Evolution Recurrent Neural Networks

Learning to Diagnose with LSTM Recurrent Neural Networks

Deep Reinforcement Learning with Double Q-Learning

Vasopressors for the Treatment of Septic Shock: Systematic Review and Meta-Analysis

Supervised Representation Learning: Transfer Learning with Deep Autoencoders

Doubly Robust Covariate Shift Correction

Adam: A Method for Stochastic Optimization

CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

Learning Factored Representations in a Deep Mixture of Experts

Domain Adaptation under Target and Conditional Shift

Epidemiology and outcomes of acute respiratory failure in the United States, 2001 to 2009: a national survey.

Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach

A Survey on Transfer Learning

Variables with time-varying effects and the Cox model: Some statistical concepts illustrated with a prognostic factor study in breast cancer

Survival analysis: time-dependent effects and time-varying risk factors.

Covariate Shift Adaptation by Importance Weighted Cross Validation

Frustratingly Easy Domain Adaptation

Analysis of Drifting Dynamics with Neural Network Hidden Markov Models

Patient Risk Stratification with Time-Varying Parameters: A Multitask Learning Approach

Definition, classification, etiology, and pathophysiology of shock in adults

Supervised Sequence Labelling

Random Forests

Noninvasive positive pressure ventilation via face mask. First-line intervention in patients with acute hypercapnic and hypoxemic respiratory failure.