Did it Work?�Using Evaluation to Understand Impact & Outcomes

State Budget Department�Grants Management Office

Mark Gordon, Governor | Kevin Hibbard, Director

​

Dru Palmer, Grants Integration Manager

Christine Emminger, Administrator for the Grants Management Office

Diana Cabriales, Deputy Administrator for the Grants Management Office

June 23, 2026

Session Objectives

Develop clear, measurable indicators that measure the success of a program’s activities

Apply a “begin with the end in mind” approach to project design by building measurable indicators into the design

Assess whether a program “worked” using qualitative and quantitative measures and program evaluation methods

​

What is an Evaluation?

Definition:

The U.S. Government Accountability Office defines an evaluation as a systematic method for assessing program design, measuring implementation, determining outcomes and impact, and informing decision making

Discussion Question

By show of hands, who has implemented a program evaluation?

Why Conduct an Evaluation?

An evaluation may be implemented for a variety of reasons. Some reasons to conduct an evaluation include:

​

Why Conduct an Evaluation ?

An evaluation may be implemented for a variety of reasons

​

Some reasons to conduct or use an evaluation

Test a theory of program change

Ensure Accountability

Assess Return on Investment (ROI)

Strengthen program management

Identify a program's outcome(s) or impact(s)

Determine program effectiveness

Types of Evaluations

There are several types of evaluations, and each serves a different purpose depending on what you want to learn about a program

Common Evaluation Terms

Goal

Objective

Activity

Big Picture - a broad, long-term aspiration that provides direction but is not measurable on its own

Specific - breaks the goal into specific, measurable outcomes that indicate progress toward that goal (“SMART”)

​

Action - a specific action or set of tasks carried out to achieve this objective

​

Specific data points used to track and measure success

Indicators

Outcomes

Program

Organized set of activities to achieve specific goals

Outputs

Impacts

Indicators

Larger Impact of “Evaluation” Data

In addition to program evaluations, designing projects to be able to collect data is also essential for:

Potential Evaluation Questions

Measurable Outcomes Scenario

Program Goal: Improve students’ academic performance in math

​

Objective: Increase the percentage of participating middle school students who achieve proficiency in math by 20% over the academic year

​

Activity: Provide 3 weekly tutoring sessions

​

Indicators:

• Number of tutoring sessions offered
• Number of students who attend tutoring sessions
• Percentage of participating students with improved math scores
• Average percentage increase in math test scores among participants

​

After-School Tutoring Program

Baseline and Target Data

For each indicator, programs will identify a baseline and target

Selecting Indicators to Capture Meaningful Change

Combine baseline, early indicators, and long-term outcomes to detect hidden and growing program impact

Key Takeaway

Baseline data is important! Show where participants start; without it, progress may seem invisible

​

Early indicators are be modest, yet progressive signs to measure whether the program is on track to achieve its intended results

​

Strong indicators are relevant – you can count or observe it, and measurable – connected to the program’s goals.

1

2

3

Capturing Incremental Change

Indicators can be gradual. Some examples of indicators that capture meaningful and gradual change include: 

Measure of positive progress (e.g., skills, confidence, participation)​

Measure of avoided negative outcomes (e.g., absenteeism, dropouts)​

Small changes that predict bigger outcomes ​(e.g., 1st step, joining a club; up to assuming leadership roles)

Improvement Indicators

Prevention Indicators

Early/Leading Indicators

Set-Up for Success: Begin with the End in Mind

Building Projects With the End in Mind

Project Design Questions:

1. How will we know if this program was successful?
2. What do we hope to achieve?

​

Throughout Implementation Questions:

1. Does our data show that we are working towards success?
2. Are we serving our intended population?
3. Are we on track to achieve a positive return on investment?

Evaluation Questions:

1. Did it work?
2. Who was served?
3. What was our return on investment?

Evaluation doesn’t have to be formal. Project managers can ask these questions along the way.

Tip

How to Build with the "End in Mind"

What steps can project leads take at the beginning of a project to position themselves to conduct an evaluation? 

​

Map the Path

Build a comprehensive Logic Model for the project. This will connect your mission and goals to measurable indicators.

Define Success

From that logic model, define what success can look like – from capacity building outcomes to service outcomes.

Ask the Right Questions

How is data going to be reported, how often, and by whom?

Involve Subject Matter Experts for �Technical Knowledge

Capacity Building 

Policy and Research Collaboration

Technical Partnerships 

Train government staff to understand and use data and evidence

Connect experts, data scientists, and policy practitioners

Partner with universities, think tanks, or nonprofits for evaluation capacity

Common Challenges in Evaluation

Unclear outcomes

Lack of baseline data

Activities and impacts align poorly

Insufficient data collection tools

Limited resources

Evaluation Activity

Activity

22

Instructions: Working in small groups, use the Did it Work? Evaluation Session Worksheet to develop an evaluation framework for a project�

Activity Steps: After reviewing the scenario with the Instructors, groups will:

• Step 1: Brainstorm how you would demonstrate success 
• Step 2: Identify evaluation approach
• Step 3: Identify potential indicators  
• Step 4: Identify data needs and consider organizational capacity

Did It Work? Measuring Program Effectiveness

So, You Think You Want to Conduct an Evaluation...

There are five steps to consider when conducting an evaluation:

Step 1: What is the scope of this evaluation?

Step 2: What data will I need to collect to answer those evaluation questions?

Step 3: How will I analyze and interpret my data?

Step 4: How will I communicate or report the findings?

Step 5: How can I use evaluations long-term for continuous improvement?

Step 1: Define Evaluation Scope

1

2

3

4

Step 2: Collect Evaluation Data

Ways to Collect Data

• Surveys
• Interviews
• Focus Groups
• Administrative Data
• Observations
• Tests or Assessments

Where Data Comes From

Internal Sources:

• Enrollment forms
• Attendance logs
• Case management systems

External Sources:

• Government databases
• Partner organizations
• Public datasets

​

Where Data Is Housed

Simple Setups

• Excel Spreadsheets, Survey platforms (E.g., SurveyMonkey)

Data Systems

• Databases,
• CRM systems 
• Secure cloud platforms

Step 3: Analyze Evaluation Data

The type, structure, and quality of data determines what you can analyze — and how confidently you can act on the results.

​

Characteristics of a Quantitative Analysis

Quantitative data measures numerical data and provides measurable evidence of change. Characteristics include:

​

Pre- and post-tests

Surveys with scaled responses

Attendance and completion rates

Performance metrics

Statistical comparisons

Characteristics of a Qualitative Analysis

Qualitative data explores purpose and experiences and provides depth and context behind the numbers. Characteristics include:

Interviews

Observations

Case studies

Focus groups

Open-ended survey responses

Data Analysis

Take the following steps to analyze data:

Making Qualitative Data Quantitative

When only qualitative data is available, program teams can convert words or themes into numbers so they can be analyzed statistically.

Mixed-Method Evaluation

Quantitative Data

Shows what changed

Qualitative Data

Explains why and how

Mixed-Method

Together, they provide a fuller picture of impact

Mixed-method evaluations combine both quantitative and qualitative data to gather understanding that either method could not provide alone

Interpreting “Did It Work?”

A program “worked” if:

Outcomes were achieved

Positive change is demonstrated

Evidence supports impact

Results justify resources invested

Also consider:

• What needs improvement?
• Counterfactual data: Probable outcomes without the intervention

Step 4: Communicating Findings

Visualization Tools

Transparent Reporting

Storytelling with Evidence

Use dashboards, infographics, and summaries to make complex data understandable

Present findings in clear, actionable formats for policymakers and the public

Translate quantitative data into real-world impact narratives

1

2

3

Step 5: Evaluation for Continuous Improvement

​

​

​

Build a culture of continuous learning

Continuous improvement supports:

• Funding Justification: Findings to aid current and future budget decisions
• Program Refinement: Making real-time adjustments to policy or program design
• Knowledge Sharing: Creating repositories or dashboards for lessons learned
• Learning Culture: Incentivizing innovation and reflection rather than failure

Evaluation Resources

There are credible program evaluation resources you can use for research and building evaluations:

​

• American Evaluation Association - https://www.eval.org - The AEA is devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. The website has evaluation journals and recorded webinars.

​

• U.S. Department of Health & Human Services (HHS)
• HHS Evaluation Plan - final-fy-2022-evaluation-plan.pdf

​

Evaluation Resources

• U.S. Government Accountability Office
• Designing Evaluations (GAO Guide)
• GAO-13-570 Program Evaluation Report
• GAO Report on Evaluation Methods and Evidence
• GAO-03-454 Evaluation Capacity Report
• GAO-11-176 Evaluation Planning Report

​

• U.S. Centers for Disease Control and Prevention
• CDC Program Evaluation Action Guide

​

​

State Budget Department�Grants Management Office

Email: sbd-grants@wyo.gov 

Website: https://sbd.wyo.gov/grants

​

Kevin Hibbard, Director, Wyoming State Budget Department

Christine Emminger, Administrator for the Grants Management Office

Diana Cabriales, Deputy Administrator for the Grants Management Office

Email: dru.palmer@wyo.gov

Dru Palmer, Grants Integration Manager

Governor’s Office�Grants Management Initiative

Hagerty Consulting, Inc