Nested ensemble machine learning to predict heart attack risk in patients with chest pain

Chris Kennedy

Kaiser Permanente Division of Research

PhD candidate in biostatistics at UC Berkeley

Context for why I'm here

My background / plans

• PhD candidate in Biostatistics
• Expecting to file dissertation in July
• Moving to Boston around July, depending upon pandemic status
• Looking to find a postdoc with a biomedical deep learning focus

Dissertation chapters

• Nested ensemble machine learning to predict heart attack risk in patients with chest pain
• Multitask, ordinal deep learning and faceted Rasch modeling for debiased, explainable, interval hate speech measurement (hatespeech.berkeley.edu)
• Possible biomedical extensions: analysis of imaging for cancer severity grading
• Funding targets: NSF, industry, foundations
• Discovering toxic exposure mixtures and ranking variable set importance via cross-validated causal inference and machine learning
• Related to research by Chirag Patel, Francesca Dominici
• Funding targets: NIH, EPA

Methods interests

• Deep learning for text, imaging, time series, waveform (e.g. MIMIC), audio
• Supervised, unsupervised, reinforcement
• Machine learning
• Hyperparameter optimization, ensemble meta-learning, feature selection, clustering, GLRM
• Causal inference (targeted learning)
• Exposure mixtures, causal variable importance, optimal treatment regimes, nonparametric dose-response
• Randomized trials
• Adaptive design, machine learning for covariate adjustment, rerandomization
• Item response theory / psychometrics

​

Returning to the chest pain project

Collaborators

• Mary Reed, DrPh
• Dustin Mark, MD (CREST - https://www.kpcrest.net/)
• Jie Huang, PhD

Caveat: preliminary results

Please do not cite or share results.

Preprint in the works and planned for release by July.

Scientific question (exploratory)

Among patients who present to KP with chest pain, what is their probability of having a heart attack or other major adverse cardiac event (MACE) in the next 2 months?

​

This risk estimate could potentially support improved resource allocation/workup and patient outcomes/discharge, based on low-risk cut-off of < 0.5% (or 1% or 2%).

• Legal risk is thought to contribute to overtreatment of patients who present with chest pain in the emergency department

Topics to cover

• Background on chest pain
• Missing data
• Imputation with generalized low-rank models (GLRM)
• Machine learning
• Variable importance
• Accumulated local effect plots
• Exploratory data analysis

Data structure

Observations: 116,764

Outcome: major adverse cardiac event (MACE) in 60 days (1.88% positive)

Covariates (65):

• Troponin - muscle protein that indicates heart injury
• Benchmark risk scores: EDACS, HEART
• Treadmill stress test results, history of coronary artery disease or heart attack
• Age, gender, race, BMI, blood pressure, smoking status, lipid profile, kidney GFR
• Text search of clinical notes: ECG interpretation, pain features (radiating, stabbing, palpation, inspiration), diaphoresis, exertional symptoms
• Missingness indicators

​

Missing data

ggplot, kableExtra

Correlation structure of missingness

superheat::superheat()

Generalized low rank models for imputation

• GLRMs extend PCA to arbitrary loss functions, regularization/sparsity, and non-numeric data types (binary, ordinal, categorical)
• Unification of non-negative matrix factorization, matrix completion, sparse and robust PCA, k-means, k-SVD, and maximum margin matrix factorization
• Project original dataset to (e.g.) 10-dimension low-rank approximation, then transform back to the original scale to impute missing values in a matrix completion style.
• Udell, M., Horn, C., Zadeh, R., & Boyd, S. (2016). Generalized low rank models. Foundations and Trends® in Machine Learning, 9(1), 1-118.
• Implementations currently available in h2o.ai and Julia
• Importance of hyperparameter tuning

Hyperparameter tuning for GLRM: 5 dimensions

• Number of archetypes (components)
• Range from 1 to number of covariates
• Amount of regularization on X (compressed dataframe)
• Range from 0 to infinite
• Type of regularization on X
• L1 (sparse) or L2 (denoised)
• Amount of regularization on Y (feature weightings to define archetypes)
• Again 0 to infinite
• Type of regularization on Y
• L1 (sparse) or L2 (denoised)

These can be optimized using a training/test split (e.g. 80/20) or cross-validation

First round of hyperparameter tuning: 300,000

Default hyperparameters with 19 components: 787,000

3rd round of hyperparameter tuning: 2,000

Optimal settings for missing value imputation

• k = 50 (versus 57 original covariates)
• Regularization amount for X: 4
• Regularization type for X: L2 (dense denoising)
• Regularization amount for Y: 24
• Regularization type for Y: L1 (sparse denoising)

Imputation error comparison: GLRM vs. Median/Mode

GLRM: examining cumulative variance explained

Machine Learning

Decision tree baseline (6 leaf nodes)

Given class imbalance, it was critical to re-weight the loss matrix for inverse class proportion.

​

Alternatively one could use observation weights, but then the plots can't accurately show the proportion of dataset in each leaf node.

Performance metrics

• Primary: area under the precision-recall curve (PR-AUC)
• Precision: correctly predicted positive / predicted positive
• TP / (TP + FP)
• "positive predictive value"
• Recall: correctly predicted positive / all positive observations
• TP / (TP + FN)
• "sensitivity" or "true positive rate"
• Note that it does not include a true negative term (TN)
• Calculate precision vs. recall at all possible thresholds to get PR-AUC
• "average precision"
• Baseline PR-AUC is the outcome average (percentage of positive cases)
• Secondary: area under the ROC curve (ROC-AUC)
• True positive rate: recall TP / (TP + FN)
• False positive rate: 1 - specificity
• FN / (TN + FN)
• Baseline AUC is 0.5
• With imbalanced data AUC will be high due to negative cases being classified correctly
• Intuition: % of time that random positive case ranked higher than random negative case

precrec, cvAUC

Discrimination: PR-AUC

1.88%

Discrimination: ROC-AUC

ROC Curve

SuperLearner weight distribution

Random Forest plateau analysis

Calibration

Calibration: overall

Calibration: zoomed in and exponential scale

Variable importance

• How influential or impactful is each variable on the predictive accuracy of our models?
• For a given model, which variables are driving the predictions?
• Methods
• Correlation (shown in EDA)
• Single decision tree
• OLS
• Random forest
• XGBoost
• To add: BART, vimp, permutation importance

Vimp: Decision Tree

vip, rpart

Vimp: OLS

vip

Vimp: Random Forest

vip, ranger

Vimp: XGBoost

vip, xgboost

Accumulated local effect plots

• Show how covariate influences prediction
• Examines change in prediction for a certain variable when it is adjusted over a small range
• Ex: for observations with age between 50 and 60, what is the average prediction when we set all those patients to age 60 compared to age 50?
• Improves on partial dependence plots, which make unrealistic assumption that features are not correlated
• See Chapter 5 in Interpretable Machine Learning

ALE: troponin peak, age, EDACS

iml

ALE: troponin 3-hour, HEART, pulse

iml

Exploratory data analysis

• Analyze key continuous variables
• Density of variable separately for 0's vs. 1's
• Smoothed risk across range of variable
• Compare to baseline risk
• Plot linear correlation

EDA: EDACS

EDACS: inefficient resolution of score

EDA: age

EDA: BMI

Correlation (Pearson)

ck37r::

vim_corr()

Thanks - any questions, comments, or feedback?

​

​

Twitter: @c3k

Website: ck37.com

GitHub: github.com/ck37