1 of 32

Medication for Opioid Use Disorder, Predictability of Retention vs. Attrition

Final Study Report CTN-0136

Jonathan Chen, M.D, Ph.D. and Fateme Nateghi, Ph.D./ Stanford

Alex Vance, M.S., L.M.H.C./Holmusk

2 of 32

Overview

OUD major public health issue, affecting millions of people worldwide

buprenorphine-naloxone and methadone are proven to help reduce opioid-related deaths

The biggest challenge is keeping patients in treatment. Almost 70% attrition

3 of 32

How does it work in practice?!

Additional counseling

Medication adjustments

Targeted support services

4 of 32

Web App Interaction

5 of 32

Web App Interaction

6 of 32

Predictive Model Opportunity?

Predict death in terminal/ICU patients? – Actionable?
Administer oxygen for hypoxic patient? – Arbitrary?
Diversion of prescribed opioids? – Ascertainable?
Allocate a Scarce Resource or Risky Treatment

E.g., Hospitalization, Clinical Trial Enrollment, or Medication

Actionable	Arbitrary	Ascertainable
Viable Important Decision	Variable Human Prediction	Verifiable Correct Answer

7 of 32

Supervised Learning, Classification

Heart Rate

Oxygen Saturation

8 of 32

Supervised Learning, Classification

Heart Rate

Oxygen Saturation

9 of 32

Supervised Learning, Classifiers

http://scikit-learn.org/stable/auto_examples/classification/plot_classifier_comparison.html

Input Data

Nearest Neighbors

SVM -

Linear

SVM -

Radial Basis

Gaussian Process

Decision Tree

Random Forest

Neural Net

AdaBoost

Naïve Bayes

10 of 32

Scan to download the paper

in this research we Predict 6-month retention in buprenorphine-naloxone (BUP-NAL) treatment using EHR data.

While keeping patients in treatment can be pretty complex and varies a lot, being able to tell which patients are high-risk and which are low-risk using machine learning means we can tailor individual management and follow-up strategies better.
in adidiiton to predicting high-risk, and low risk patient, we also
Identify key predictors influencing treatment attrition.

Data Sources:
Stanford Single-Center Dataset:

Origin: Data sourced from STAnford Research Repository (STARR)
Stanford data include visits from 1993 to 2022
1,800 treatment encounters

NeuroBlu Multicenter Dataset:

Origin: Acquired from Holmusk, a behavioral health real-world data company
included 20+ years (2003 - 2023) of longitudinal de-identified patient-level clinical data from patients across all 50 US states who received treatment from 166 distinct sites of care, including psychiatry-specialty and multispecialty sites

11 of 32

STARR-OMOP Data sources

Stanford Healthcare Dataset:

Visits from 1993 to 2022 who received BUP-NAL
1,800 treatment encounters

NeuroBlu/Holmusk Multicenter Dataset:

Behavioral health real-world data company
20+ years (2003 - 2023) across 50 states
7,957 treatment encounters

12 of 32

STARR-OMOP Extracted Features

BUP-NAL

End Data

Treatment duration= 89 days

BUP-NAL Start Date

ML Prediction

Condition

Drug

Procedure

Condition

…

189 variables extracted from EHR + demographics

13 of 32

Patient Timeline

14 of 32

High-Risk Vs. Low-Risk Patient?

15 of 32

Automatic Chart Review

PTSD

major depression

homelessness

ADHD

bipolar disorder

SUD

Tobacco dependence

chronic pain

suicidal behavior

alcohol dependence

liver disease

unemployment

clinical notes

personality disorder

16 of 32

Large Language Models (LLMs)

https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html

17 of 32

Concept Extraction

18 of 32

CLinical Entity Augmented Retrieval (CLEAR)

CLEAR pipeline

Clinical notes

19 of 32

How CLEAR Works?

20 of 32

How CLEAR Works?

21 of 32

How CLEAR Works?

22 of 32

How CLEAR Works?

Scan to download the paper

23 of 32

CLEAR Validation Results

24 of 32

Model Performances

Source	Model	Precision	Recall	ROC-AUC
Stanford	Logistic Regression	0.69 ± 0.2	0.52 ± 0.3	0.57 ± 0.3
	Random Forest	0.70 ± 0.1	0.77 ± 0.2	0.64 ± 0.3
	XGBoost	0.72 ± 0.2	0.75 ± 0.2	0.65 ± 0.3

25 of 32

Model Performances

Source	Model	Precision	Recall	ROC-AUC
Stanford	Logistic Regression	0.69 ± 0.2	0.52 ± 0.3	0.57 ± 0.3
	Random Forest	0.70 ± 0.1	0.77 ± 0.2	0.64 ± 0.3
	XGBoost	0.72 ± 0.2	0.75 ± 0.2	0.65 ± 0.3
NeuroBlu	Logistic Regression	0.90 ± 1.0	0.20 ± 0.7	0.60 ± 0.3
	Random Forest	0.83 ± 0.1	0.78 ± 0.2	0.56 ± 0.3
	XGBoost	0.86 ± 0.1	0.56 ± 0.3	0.57 ± 0.3

26 of 32

Stanford and Holmusk Data Demographics

Feature	Stanford Dataset (%)	NeuroBlu Dataset (%)
Age
Mean (SD)	49.4 (16.5)	42.2 (12.4)
Median (IQR)	51 (28)	40 (18)
Sex
Male	52.8%	59.3%
Female	47.2 %	40.7%
Race
White	72.1%	43.4%
Black or African American	6.6%	55.6%
Ethnicity
Hispanic or Latino	12.4%	0%
Not Hispanic or Latino	84.5%	0.4%
Attrition Rate	61%	83%

27 of 32

Shapley Values

28 of 32

Human validation

Objective: Compare model predictions to clinician predictions.

Clinicians Involved: 3 board-certified addiction medicine physicians reviewed 147 Stanford patient charts.

Predictions: Clinicians predicted 6-month treatment retention

29 of 32

ML Models Vs. Human

Source	Model	Precision	Recall	ROC-AUC
Stanford	Logistic Regression	0.69 ± 0.2	0.52 ± 0.3	0.57 ± 0.3
	Random Forest	0.70 ± 0.1	0.77 ± 0.2	0.64 ± 0.3
	XGBoost	0.72 ± 0.2	0.75 ± 0.2	0.65 ± 0.3
NeuroBlu	Logistic Regression	0.90 ± 1.0	0.20 ± 0.7	0.60 ± 0.3
	Random Forest	0.83 ± 0.1	0.78 ± 0.2	0.56 ± 0.3
	XGBoost	0.86 ± 0.1	0.56 ± 0.3	0.57 ± 0.3
3 board certified addiction medicine clinicians		0.70	0.60	0.68 ± 8.9

30 of 32

Conclusions

-Bup-Nal treatment retention Moderately Predictable

-Natural Language Processing (NLP) for Contributing Factors

-(e.g., depression, liver disease)

-Incremental Accuracy Gains with NLP

-Quantitative Tools and a Reusable Web Apps available

-(e.g., clinical practice, risk adjusted quality measures)

31 of 32

Publications from CTN-0136 Study

Published Papers:

Steven Tate et al. The ChatGPT therapist will see you now: Navigating generative artificial intelligence’s potential in addiction medicine research and patient care. (Addiction, 2024)
Ivan Lopez et al. Predicting premature discontinuation of medication for opioid use disorder from electronic medical records. (AMIA Annual Symposium Proceedings 2023)
Fateme Nateghi Haredasht et al. Predicting Treatment Attrition in Buprenorphine-Naloxone Therapy: A Machine Learning Approach Using Multi-Site EHR Data. (AMIA Annual Symposium Proceedings 2024)
Fateme Nateghi Haredasht, Sajjad Fouladvand, Steven Tate et al. Predictability of Buprenorphine-Naloxone Treatment Retention: A Multi-Site Analysis Combining Electronic Health Records and Machine Learning. (Addiction, 2024)
Ivan Lopez, Fateme Nateghi Haredasht et al. Clinical entity augmented retrieval for clinical information extraction. (npj Digital Medicine, 2024)

Upcoming Submission:�Fateme Nateghi Haredasht, Ivan Lopez et al. Predicting Treatment Retention in Medication for Opioid Use Disorder: A Machine Learning Approach Integrating LLM-Derived Clinical Features. (To be submitted)

32 of 32

Acknowledgements

Students + Collaborators
NIH/National Institute on Drug Abuse Clinical Trials Network (UG1DA015815 - CTN-0136)
NIH/National Institute of Allergy and Infectious Diseases (1R01AI178121-01)
NIH-NCATS-Clinical & Translational Science Award (UM1TR004921)
Gordon and Betty Moore Foundation (Grant #12409)
Stanford Bio-X Interdisciplinary Initiatives Program
NIH - Center for Undiagnosed Diseases (U01 NS134358)

Content does not necessarily represent the views of any agency or institution

Financial disclosures: Reaction Explorer LLC – Co-founder. Expert witness / Consulting / Speaking fees from Younker Hyde MacFarlane, Sutton Pierce, Sykes McAllister, ISHI Health, Cozeva