1 of 1

Measuring Implementation to Impact: Evaluation of a Chronic Pain Integrative Medical Group Visit Program

Chronic pain, opioid misuse, and social isolation are public health crises. Current clinical practice guidelines recommend non-invasive mind-body therapies as a first-line treatment for chronic pain, and studies indicate these therapies can be successfully delivered in group settings using an Integrative Medical Group Visit model.

A Federally Qualified Health Center in rural California piloted a nurse-led Integrative Chronic Pain Group (ICPG) program for a portion of their patients experiencing chronic pain and receiving medication for opioid use disorder.

Quality (Execution)

Value (Cost-Effectiveness)

Significance (Outcomes)

The organization exhibited inherent strengths necessary to ensure that the quality, value, and significance of the program are maintained and enhanced, including dedicated providers, executive level buy-in, appropriate physical facilities, a large target patient population, electronic health records for data gathering, and a desire to provide value-based care. Application of the CDC evaluation framework created a progress report toward established goals and revealed quality improvement opportunities.

Though strong conclusions about significant program-related effects could not be drawn because of the lack of program compliance data and low statistical power, the ICPG did demonstrate alignment with the core principles of value-based care outlined by the IHI and the IOM; it increased access to integrative chronic pain and opioid use disorder management in a social setting, it was feasible and sustainable to implement, and it positively impacted patients and providers. Therefore, the ICPG program has the potential to positively impact local public health by addressing chronic pain, opioid use disorder, and social isolation for vulnerable patients of a rural, safety-net clinic.

Dr. Mercedes Echevarria, primary project advisor

Dr. Bill Hunter, secondary project advisor

A special thank you to all the program stakeholders who donated their time and effort to provide evaluation data

Contact: pscher@gwu.edu

The Centers for Disease Control and Prevention (CDC) Framework for Program Evaluation in Public Health was chosen to guide this scholarly project.

Step 1: Stakeholders were identified via a SWOT analysis and engaged in the evaluation plan.

Step 2: A comprehensive program description was created from stakeholder interviews and program documents, culminating in the creation of global and detailed logic models and a driver diagram.

Step 3: The models were used to focus the evaluation and assign objectives and indicators to each evaluation aim: quality (execution), value (cost-effectiveness), and significance (outcomes/impact).

Step 4: Quantitative and qualitative data were gathered via de-identified clinical and financial data, program and organizational documents, stakeholder interviews, and patient and provider satisfaction surveys.

Step 5: Data were analyzed and synthesized. Descriptive statistics and paired and independent t-tests were completed, program attendance and compliance were calculated, revenue values were estimated, program fidelity was analyzed, and best practice alignment was determined by comparing the ICPG program to two successful chronic pain Integrative Medical Group Visit models. Conclusions were drawn by interpreting the evidence through the lens of stakeholder values and benchmarking it against the Quintuple Aim of Healthcare Improvement and the Six Domains of Healthcare Quality.

Step 6: A formal report and presentation were created and presented to key program stakeholders.

Paula Tomczak, DNP, RN, LMT, CPH, CNE; Mercedes Echevarria, DNP, ACN, CNE; Bill Hunter, MD

The purpose of evaluating the ICPG program was to provide actionable, evidence-based information to inform stakeholder decision-making and improve program effectiveness.

Formal evaluation also allowed the program to be reviewed against the standards of the Institute for Healthcare Improvement’s (IHI) Quintuple Aim for Healthcare Improvement, and the Institute of Medicine’s (IOM) Six Domains of Healthcare Quality to determine alignment with the core principles of value-based care.

Several process, effectiveness, efficiency, cost, and outcomes measures were analyzed and interpreted through the lens of program stakeholders to evaluate the ICPG program’s:

quality (execution)
value (cost-effectiveness)
significance (impact)

Objectives

Background

Methods

Results

Conclusions

Acknowledgments

Recommendations

Quality (Execution)

Offer drop-in groups
Provide group facilitator training to providers
Include complementary healthcare providers
Offer dedicated program improvement time

Value (Cost-Effectiveness)

Bill for Medicare General Care Management
Understand and address attrition
Clarify the business case

Significance (Outcomes/Impact)

Strengthen monitoring and evaluation
Offer mobile groups
Leverage environmental assets

Program implementation and execution demonstrated fidelity to the original plan and aligned with best practices established by previous IMGV programs, except that complementary healthcare practitioners were not employed (Figure 2)

Value and sustainability indicators revealed the program generated slightly more annual revenue than individualized care, and scalability may be possible given the demographic alignment of program participants with the larger organizational population

(Table 1)

Clinical outcomes detected a significant decrease in PHQ-9 depression scores and a marginally significant decrease in SBP (Table 2)

Satisfaction survey scores indicated considerable satisfaction with execution and impact (Table 3)

Qualitatively, peer support and group involvement were commonly cited as benefits

Figure 2

Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999;48 (No. RR-11)

Figure 1

Good afternoon and thank you for still being here! I am Paula Tomczak, and I am very happy to be here presenting my DNP project that I completed under the guidance of Dr. Echevarria: Measuring Implementation to Impact: Evaluation of a Chronic Pain Integrative Medical Group Visit Program. I wanted to take a moment to acknowledge that the region of Northern CA where I completed my work is on the ancestral and present homeland to several Indigenous nations including Hupa, Karuk, Mattole, Tolowah, Wailaki, and Yurok Peoples.

Chronic pain, opioid misuse, and loneliness and social isolation are current public health epidemics. A rural California Federally Qualified Health Center, located in a health professional shortage area for both primary care and mental health providers, recognized that their patients were significantly impacted by these crises and identified Integrative and Collaborative Care as an organizational priority. They knew that current clinical practice guidelines recommend non-invasive, mind-body therapies as a first-line treatment for chronic pain, and these therapies could be successfully delivered in a group format. In the Fall of 2022 they implemented a Nurse Practitioner-led Integrative Chronic Pain Group program for some of their patients with both chronic pain and opioid use disorder, and I conducted a program evaluation of their initiative to provide them with a holistic progress report and actionable quality improvement recommendations.

I followed the CDC Framework for Program Evaluation in Public Health to assess the program’s quality, value, and impact. I chose this framework because the CDC states that a high-quality evaluation can improve population health outcomes by improving clinical practice and it allowed me to apply the 10 Essential Competencies for Professional Nursing Education while being IRB-exempt. I also recognized that there is often a lack of funding and resources allotted to evaluating programs in safety net clinics, which makes it difficult to determine whether a program is actually meeting its stated goals.

I evaluated the program by collecting and analyzing several quantitative and qualitative indicators, interpreting them through the lens of stakeholder values, and benchmarking them against the IHI and IOM value-based care standards. Demographic, financial, and screening tool data came from de-identified electronic health record reports; patient and provider satisfaction data were collected through anonymous surveys; and program implementation data were collected through structured interviews and organizational documents.

Through application of descriptive statistics, paired and unpaired t-tests, basic cost analysis, and comparison of internal and external program fidelity and alignment, I found that program execution was faithful to the initial plan and aligned with many best practices recommended by established integrative medical group visit programs. Value indicators revealed that the program was feasible to implement and generated slightly more annual revenue than one-on-one care, and demographic alignment of program participants with the larger target population suggested that the program could possibly be scaled within the organization. Clinical indicators detected a significant decrease in PHQ-9 scores measuring the severity of depression, and a marginal decrease in systolic blood pressure levels, which were shared savings metrics identified by their accountable care organization partner. Participant and provider satisfaction surveys indicated great satisfaction with the execution and impact of the program, and identified peer support and group involvement as significant program benefits.

I then turned the findings into actionable recommendations, which I organized according to each aim, which included

hiring complementary healthcare providers, offering groups in an open format, providing group facilitator training to providers, and allotting dedicated program-improvement time to improve program quality
Billing new Medicare chronic pain management codes, creating a virtual attendance option, and providing dedicated support staff to complete auxiliary health maintenance activities, to improve cost-effectiveness
And strengthening monitoring and evaluation, offering mobile groups, and leveraging natural environmental assets to improve program impact

In conclusion, program evaluations aren't meant to produce generalizable knowledge, and though my evaluation had many limitations, including a lack of robust financial and clinical data, and a low power to identify significant program-related effects, the program, co-facilitated by a nurse practitioner and licensed clinical social worker, did demonstrate alignment with the core principles of value-based care defined by the Institute for Healthcare Improvement and the Institute of Medicine: it increased access to safe, cost-effective, patient-centered, chronic pain care management for vulnerable patients of a rural, safety net clinic, while also improving patient and provider satisfaction and clinical outcomes.