1 of 26

Parkland Health Augmented Intelligence in Medicine and Healthcare Initiative (AIM-HI): ��Generalizability of An AI/ML-Driven Asthma QI Program in Safety Net Systems

Program Team

  • Cesar Termulo, MD, Parkland Health  (PH) - Presenter
  • Yolande Pengetnze, MD., M.S., Parkland Center for Clinical Innovation (PCCI) - Presenter
  • George Oliver, MD, PhD., PCCI
  • Amrita Waingankar, MD, Parkland Community Health Plan (PCHP)
  • Naomi Gebrelul, MD, Foremost Family Health Centers (Foremost)
  • Sharon Davis, DO, Los Barrios Unidos (LBU)

​

2 of 26

National Asthma Epidemiology

  • > 4.6 million children with asthma nationally
  • > 7% of the population under 18 years old
  • >270,000 emergency department (ED) visits per year
  • 145 pediatric deaths in 2021
  • 3.8 million in the United States live under the poverty threshold
  • Highest burden among minority and low-income families:
      • By Race/Ethnicity: Black 12%, Latinos 7% vs. White 6%
      • Asthma-related ED visits are nearly 5-fold high for Black patients compared to white patients.
      • Asthma deaths are 8-fold higher for Black vs. White children

3 of 26

Community Health Needs Assessment at Parkland

  • Parkland is Dallas County's safety net system
    • 983 inpatient beds
    • >60,000 admissions/year
    • >230K ED visits/year
    • >1.1 ambulatory visits/year
    • 17 community clinics (14 with pediatric services)
    • Correctional health, 8 mobile vans (homeless outreach), 2 school-based clinics

​

    • First Community Health Needs Assessment in 2016
    • Initiative established for Pediatric Asthma
    • Also established initiatives for mental health, diabetes, breast cancer, post-partum mortality, hypertension, and STD's

​

​

4 of 26

CHNA Pediatric Asthma Zip Codes

5 of 26

Role of AI In Vulnerable Populations

5

Opportunities for AI in Improving Asthma Outcomes

The Problem:  

  • Parkland has over 2,000 children with asthma
  • Providing accessibility is a challenge with provider panels around 1800 patients per provider
  • Many SDOH issues negatively affect patient outcomes separate from quality of medical care
  • AI must drive remote monitoring our large patient population

​

Our Opportunity: 

  • Timely diagnosis of rising asthma risk is a cornerstone of asthma care and key to preventing asthma ED visits and hospitalizations. 
  • AI/ML can capture the cumulative effect of multiple, actionable risk drivers to accurately predict rising asthma risk for timely and tailored clinical decision-making.

6 of 26

AI/ML Asthma Care  �Program Description

Parkland AI/ML Asthma Risk Model Framework

PPV=Positive Predictive Value; EHR=Electronic Health Record

6

Our Asthma Care Program is Uniquely Focused on Vulnerable Populations

AI/ML Risk Prediction Model

  • Focus on outpatient and safety net settings
  • Predicts 90-day asthma ED visit or hospitalization
  • Clinical, utilization and SDOH

Monthly Risk Reports Updates

  • Risk profile sent to population health, case management, and frontline teams
  • Multifaceted and multistakeholder interventions

Clinical Workflow Integration

  • Evidence-based interventions, e.g., targeted medication management and visit scheduling, trigger mitigation, EHR integration – POC alerts

Patient Engagement Through Risk-Driven Text Messaging 

  • Education and remote symptoms monitoring

​

7 of 26

Asthma Risk Model Design

7

PARAMETERS

RATIONALE

  1. Predict risk for asthma-related ED visit or hospitalization within the subsequent three months
  • Asthma ED visits & hospitalizations costly
  • 3 months – clinically-relevant interval for effective intervention
  1. Broad definition of probable asthma – Modified CSTE* Definition
  2. Any asthma-related ED visit, hospitalization, outpatient visit or medication in the past 12 months
  • Cast broad net
  • Use risk model to stratify and sort out
  1. Include clinically-relevant, evidence-based variables
  • For clinical insights and generalizability
  1. Report on actionable risk factors
  • Drive tailored interventions

MODEL VARIABLES

  • Past and recent outpatient, ED, and Hospital Utilization
  • Controller and reliever medication use
  • Sociodemographic variables
  • Systemic steroids use
  • Medical and psychiatric comorbidities

8 of 26

Model Variables & Performance

8

Overall Performance: C-statistic 0.84 – Very Good

9 of 26

Program Implementation

Adaptable to provider priorities/specificities

​

  • Workflow Integration
      • Including EHR Point-of-Care Alerts (Parkland only)
  • Prioritized Scheduling
      • Including telehealth
  • Prioritized spirometry/allergy testing
  • Targeted CM/CHW interventions
  • Medication Management
  • Tailored Patient Education
  • Business KPIs -
      • No Shows/ Cancellation, APM Contract, Reputation / Retention

​

Monthly Patient-Level Reports

9

Reports Implementation by Providers/ CMs

10 of 26

Program Implementation

  • >80 short asthma messages
      • 2-3 messages per week

​

  • 2-item asthma symptoms
      • 1-2x/week

​

  • 3-item satisfaction survey
      • 1x/quarter

​

      • Very High, High and Medium Risk
        • (+) Anyone referred by providers

​

        • Biweekly Symptoms Survey Reports
          • To providers

​

        • Impact monitoring
          • Quantitative & Qualitative

10

Text Messaging Program & Biweekly Reports

11 of 26

Program Evaluation

PCHP (Claims-based Model): Launched in 2016

Dallas County CHNA (EHR-based Model ): Launched in 2020

​

      • Annual Outcomes Monitoring
        • Quasi-Experimental Design
        • Compare “Participants” vs. “Contemporary” controls
      • Text Messaging Impact Evaluation
        • Quantitative & Qualitative
      • Data Sources:
        • PCHP claims, Parkland EHR, and DFWHCF regional dataset
      • Outcomes of Interest
        • Primary: rolling 12-month asthma-related ED visit rates i.e., number of ED visits with asthma as primary diagnosis
        • Secondary: 12-month hospitalizations, systemic steroid use, outpatient visits, asthma meds use
        • Text Messaging program patient/caregiver satisfaction survey

Program Evaluation Timeline

CHNA=Community Health Needs Assessment; DFWHCF=Dallas-Ft. Worth Hospital Council Foundation;

11

2023-01

Program Start Date

Sliding/Rolling window

Month 1

Time

2019-07

2020-06

Month 0

Time

2019-06

2020-05

Month 12

Patient 12-month utilization history

Time

2023-12

Patient 12-month utilization history

Patient 12-month utilization history

12 of 26

AI/ML Models for Asthma Care�Historical Performance

PCHP : Launched in 2016

      • 25,000 Patients risk-stratified very year; Impact within Year 1 (Participants vs. Controls)

42% less asthma ED visits, sustained over 5 years

Data Sources: DFWHCF=Dallas-Ft. Worth Hospital Council Foundation Dataset & PCHP Claims Data;

12

32% less asthma-related costs (no controls)

13 of 26

AI/ML Models for Asthma Care�Historical Performance

CHNA Asthma Implementation Plan : Launched in 2020

13

14 of 26

AI/ML Models for Asthma Care�Historical Performance

CHNA : Launched in 2020

      • Impact within Year 1 (Participants vs. Controls)

36 - 40% less asthma ED visits, reversal of EDV trends

Data Sources: DFWHCF=Dallas-Ft. Worth Hospital Council Foundation Dataset & Parkland EHR Data;

14

        • Reversal of EDV trends

​

        • No statistically significant impact on asthma-related hospital admissions

​

        • No medication or costs data for control patients

15 of 26

AI/ML Models for Asthma Care�Historical Performance

Text Messaging Program

      • Participants vs. Controls

Data Sources: DFWHCF=Dallas-Ft. Worth Hospital Council Foundation Dataset & Parkland EHR Data;

15

16 of 26

AI/ML Models for Asthma Care�Historical Performance

Text Messaging Program

      • Participants vs. Controls

Data Sources: DFWHCF=Dallas-Ft. Worth Hospital Council Foundation Dataset & Parkland EHR Data;

16

24% less asthma-related costs within 6 months

24% less asthma-related costs within 6 months

59% less asthma-related systemic steroids use (p<.01)

17 of 26

AI/ML Models for Asthma Care�Historical Performance

Text Messaging Program

      • Qualitative Feedback - Participants only

Data Sources: PCCI Qualtrics

17

91% “Would Recommend Program to Friends and Family”

18 of 26

Text Messaging Program

  • Patient Testimonial on instagram

​

  • Scan QR Code for Video

19 of 26

Model Expansion From Parkland to Foremost & LBU�

​

​

Consort Diagram

RCT=Randomized Controlled Trials

19

From Large Public/Academic Setting to Smaller Safety Net Providers (FQHCs)

Test, Retrain, and Calibrate Model

  • Expand to other underserved communities
  • Monitor for biases and adjust accordingly

Multistakeholder Engagement

  • Clinical, population health, managed care, leadership

Rigorous Scientific Design

  • Pragmatic RCT – context-adapted
  • Intention-to-Treat and As-Treated Analyses

Clinically Relevant Impact Metrics

  • Primary Outcome: 12-month Asthma ED visit rates
  • Secondary outcomes: medication and systemic steroid use, qualitative feedback
  • Subset analyses by race/ethnicity

Long-Term Sustainability

  • Cost analysis and sustainability framework

20 of 26

Model Expansion Timeline

Two-Year Study Timeline and Milestones

20

21 of 26

North Texas PASS

Pediatric Asthma Surveillance System (PASS)

    • Community-Facing Dashboard
  • Taking AI/ML Asthma Risk Prediction to the community-level/micro-geography
    • Estimating Population-Level Asthma Risk using clinical, social and environmental risk drivers
  • PASS leveraged for Asthma 411 with Dallas Independent School District
    • Identify target communities (e.g., Southeast Dallas) and relevant partners (e.g., school campuses, medication support programs etc.)
  • PASS Article published in NEJM Catalyst
    • August 2024 edition; published July 17th

​

22 of 26

Questions/Discussions

cesar.termulo@phhs.org

​

yolande.pengetnze@pccinnovation.org

23 of 26

Appendix

24 of 26

Model Expansion

From Parkland to Foremost & LBU  

From Large Public/Academic to Smaller Safety Net Providers (FQHCs)

      • Test, Retrain, and Calibrate Model
      • Expand to other underserved communities
      • Monitor for biases and adjust accordingly
      • Multistakeholder Engagement
      • Clinical, Pop. Health, Managed Care, Leadership
      • Rigorous Scientific Design
      • Pragmatic RCT – context-adapted interventions
      • Clinically Relevant Impact Metrics
      • Asthma EDVs, medication & systemic steroid use
      • Subset analyses by race/ethnicity
      • Long-Term Sustainability
      • Cost Analysis & Sustainability framework

Randomization:

Interventions:

12-Month Outcomes:

Post-Grant Sustainability:

2 (n= 700)           1 (n= 350)

  • Monthly Risk Reports
  • EHR Alerts
  • Risk-Driven Interventions (meds management, prioritized scheduling, CHW outreach, trigger mitigation etc.)
  • Text Messaging

Intervention

Control

  • Routine Care
  • Text Messaging (optional)
  • 12-month EDVs, Meds & Steroid Use
  • Patient & Provider Satisfaction
  • Cost Analysis, Sustainability Framework, & Lessons Learned
  • Rollout to all patients
  • Technical/consulting support for value-based contracting &  funding opportunities

25 of 26

Addressing Potential �Threats to Validity

  • Cluster heterogeneity
  • Cluster generalizability
  • Recruitment
  • Endpoint capture
  • Contagion across clusters

​

​

  • Healthcare systems serve a diverse racial and ethnic mix that allows for good representation of US population
  • Model use with EHR and claims inputs allow for use in more settings
  • Powered for detection of signal below total eligible population across participants
  • Utilization of regional health care system data allow assessment in populations with varied insurance status. DFWHC Data allow for complete capture of ED and Inpatient events within local 11 county areas in addition to claims data from PCHP insured patients
  • Risk prediction integrated at Patient level with no access to risk prediction outside intervention. While providers may cover across other practices, risk model will not be available to them

26 of 26

Parkland Is Committed to Bringing Responsible AI and High-Quality Care to Vulnerable Populations�

Alignment with AIM-HI Grant Purpose and Strong Implementation Plan 

Focus on AI/ML in Safety Net Setting

  • Unique perspective of AI/ML implementation, generalizability, biases, & sustainability in traditionally marginalized communities

Model Maturity & History of Successful Expansion

  • Since 2016 (7 years), from PCHP to CHNA

Experience in Multistakeholder Program Implementation

  • Large pragmatic clinical trials experience with NIH ICD-Pieces and CMS programs

Governance Framework

​

  • Existing relationships & leadership support across stakeholders 
  • Strong stakeholder alignment

Scientifically Sound Proposal & Attainable Goals

  • Mixed design, outcomes, & robust data sources
  • Leaders in non-medical drivers of health impact measurements and program deployments

Sustainability Framework in Safety Net Settings

​

    • Cost analysis + clinical, leadership and MCO perspectives
    • Post-grant program continuation + Generalizability to safety net providers in other geographies

26

NIH=National Institutes of Health; CMS= Centers for Medicare and Medicaid Services