1 of 47

2 of 47

Designing an Evaluation - Step by Step

A Better Government Lab + Nava PBC Collaboration

Eleanor Grudin

Nava + Better Government Lab Graduate Research Fellow

Martelle Esposito, M.S., M.P.H.

Nava PBC

Eric Gianella, Ph.D.

Georgetown University

Michael Chen

Nava PBC

3 of 47

These slides were developed in a collaboration between the Better Government Lab and Nava PBC. The intention was to support training Nava staff on the importance of evaluation. We hope they are helpful to others in the Civic Tech community interested in learning about applied program evaluation.

ACKNOWLEDGEMENT

4 of 47

Contents

  1. Defining “Administrative Burdens” and “Service Outcomes”
  2. Evaluation Design and Why We Evaluate
  3. Evaluating Administrative Burdens
  4. Case Study
  5. A Real-World Example
  6. Final Survey Best Practices

4

Better Government Lab

5 of 47

The Case Study�

While loosely based on real clients, this is a fictional case study for the purposes of practicing evaluation design.

SECTION 1

6 of 47

New York Department of Labor’s Big Updates

6

Better Government Lab

The New York Department of Labor decided their online system of applying for unemployment needed a massive facelift.

7 of 47

New York Department of Labor’s Big Updates

7

Better Government Lab

A new human-centered claimant portal

An overhauled Spanish language experience

Insourcing ID.me verification

This included three major changes to their service delivery system:

8 of 47

Your Evaluation’s Scope

You have been brought on under the following parameters:

  1. The study is to be completed over the course of the roll-out.
  2. You have funding for 0.25 FTE.
  3. The client would like some evidence during the roll-out that the changes are producing the intended effects.

8

Better Government Lab

9 of 47

New York Department of Labor’s Data

9

Better Government Lab

At the beginning of the project, you learn you have access to the following data:

10 of 47

Evaluation Design

SECTION 2

11 of 47

Evaluation Design Roadmap

Identify the outcomes of interest.

Step 1

Decide on a data collection strategy.

Step 2

Develop a timeline.

Step 3

Build data collection capacities alongside the product.

Step 4

Collect and check the data.

Step 5

Analyze the data.

Step 6

Share the results.

Step 7

11

Better Government Lab

12 of 47

Identifying Outcomes of Interest

Step 1

13 of 47

Introduction to Logic Models

13

Better Government Lab

The product or service that is built to address a specific problem.

The behavior result of the intervention. This is the activity performed by the end-user.

The intervention achieves all the functional goals that the product team set out to accomplish.

The specific, directional change in customer experience that results from the intervention.

The specific directional change in the overarching goals of a group that we and the beneficiaries want to achieve.

Long-term changes to people’s lives like improved health, economic, and well being outcomes.

Intervention

Output

Product Outcome

Service Outcome

Program Outcome

Impact Outcome

14 of 47

Case Study Example: NY DOL

14

Better Government Lab

Intervention

Output

Product Outcome

Service Outcome

Program Outcome

Impact Outcome

The new website system update.

Applicants will use the new portal system and the accompanying features.

Poverty rates in Spanish-speaking communities improve.

Using the information on the previous slide, try to fill in the blanks on this logic model.

15 of 47

Case Study Example: NY DOL - Answers

15

Better Government Lab

The new website system update.

Applicants will use the new portal system and the accompanying features.

The new portal is user-friendly, with a Spanish language experience option and ID.me.

People’s self-reported administrative burden ratings improve.

Time to complete application is reduced.

Frustration scores are lower.

Claim approval rates improve.

The disparity in claim approval rates for Spanish speakers from English speakers decreases.

Poverty rates in Spanish-speaking communities improve.

Intervention

Output

Product Outcome

Service Outcome

Program Outcome

Impact Outcome

Using the information on the previous slide, try to fill in the blanks on this logic model.

16 of 47

Decide on a Data Collection Strategy

Step 2

17 of 47

Considerations for a Data Collection Strategy

  1. Determine how to measure the outcomes
  2. Identify constraints
  3. Explore implementation design options

​

  • How rigorous does your evaluation need to be?
  • What are your options?
    • 1 group, change over time
    • 2+ groups, change over time
    • Staggered roll-out
    • Randomization (e.g., A/B testing)

​

​

17

Better Government Lab

18 of 47

Case Study Example: NY DOL

How would we want to measure this? �

What variables do we need?

Where is this data currently stored?

​

Can we access it?

18

Better Government Lab

The NY DOL is very focused on the outcome of reducing application completion time. Let’s focus on this outcome as we decide on the best data collection strategy.

19 of 47

Case Study Example: NY DOL Answers

19

Better Government Lab

The NY DOL is very focused on the outcome of reducing application completion time. Let’s focus on this outcome as we decide on the best data collection strategy.

Time to completion in the old version

Time to completion in the new version

If we use a two-group design, we can compare time-to-completion between the new and old versions.

We need to build the capacity to tell who is using which version.

We have access to this data already!

20 of 47

Review the Plan with Appropriate Stakeholders

20

Better Government Lab

21 of 47

Considerations for an Engineer Review

  1. Introduce why you are evaluating! Excite the team for the very important work.
  2. Data instrumentation and storage.
  3. Feasibility and other options for getting this done.
  4. Randomization is important - are there ways we can be creative about it? (including as part of phased rollout)
  5. If required level of effort is too high, talk to eval about where to compromise.

21

Better Government Lab

22 of 47

A Deeper Dive into Data

Data collection can be one of the most daunting parts of evaluation. Here are several things to consider:

  • What data do we have access to already?
  • What data will we need the partner to share with us?
  • What data do we need to build capacity to collect?
  • What Personal Identifying Information (PII) is needed for this project, if any?
    1. How can we work around this?

22

Better Government Lab

23 of 47

Case Study:� NY DOL

Language Preference

Time to Completion

Outcome of Application

Experience Survey Results

Version of Application

Call Center Volumes

You have established based on the outcomes of interest that the variables on the right are needed for evaluations.

​

Consider which variable you would assign to the following categories:

​

  1. Data that you have easy access to through the application portal.
  2. Data that you may need to ask the NY DOL to share with you.
  3. Data you will need to build the capacity to collect.

23

Better Government Lab

24 of 47

Case Study:� NY DOL - Answers

Language Preference

Time to Completion

Outcome of Application

Experience Survey Results

Version of Application

Call Center Volumes

You have established based on the outcomes of interest that the variables on the right are needed for evaluations.

​

Consider which variable you would assign to the following categories:

​

  • Data that you have easy access to through the application portal.
  • Data that you may need to ask the NY DOL to share with you.
  • Data you will need to build the capacity to collect.

24

Better Government Lab

25 of 47

A Final Consideration - Stopping Conditions

At this point, it feels like we have been focused on building up to the evaluation. But one of the important steps in considering data collection is deciding on your stopping conditions…

​

  • Are you going to collect data until you reach a certain number of respondents?
  • Are you going to collect data for a certain time?
  • If you notice your new program or product is resulting in more psychological stress, how long do you go before shutting it down?

25

Better Government Lab

26 of 47

Case Study Example - NY DOL Stopping Conditions

Which of the following would be the best stopping condition for evaluating changes to the NY DOL unemployment benefits application process?

​

  1. 1,000 people have applied to either the new or old system since launch day.
  2. 1,000 people have applied using only the new system.
  3. 1,000 people who are in a particular group have used the system.
  4. 6 months after the launch of the new system.
  5. 6 months after the final phase of the roll-out.

26

Better Government Lab

27 of 47

Case Study Example - Answers

Which of the following would be the best stopping condition for evaluating changes to the NY DOL unemployment benefits application process?

​

  • 1,000 people have applied to either the new or old system since launch day.
  • 1,000 people have applied using only the new system.
  • 1,000 people who are in a particular group have used the system.
  • 6 months after the launch of the new system.
  • 6 months after the final phase of the roll-out.

27

Better Government Lab

All of the answers above are realistic stopping conditions, depending on which outcome is the priority.

28 of 47

Summary of Data Collection Strategy Considerations

Is causal relationship important?

What is already easy to access?

What do we need to build access to?

How do we start and stop the evaluation?

Rigor of Evaluation

Current Data Storage System

New Data Needed for Evaluation

Roll-out and Stopping Conditions

28

Better Government Lab

29 of 47

Develop a Timeline

Step 3

30 of 47

Considerations for Timelines

The ability to do an evaluation is dependent on certain timeline parameters:

​

  • The entire project timeline
  • The workload to develop new data collection capacities
  • Funding and the amount of labor individuals are allocated
  • The rigor of the evaluation

30

Better Government Lab

31 of 47

Build Data Collection Capacities Alongside the Product

Step 4

32 of 47

Data Collection & Storage Capacities

32

Better Government Lab

This is an example of data mapping and creating a plan for data storage. This hopefully will be executed during the product development so the evaluation can proceed.

33 of 47

Begin Collecting Data and Check on Preliminary Results

Step 5

34 of 47

True or False?

You should begin collecting data right after the product launches.

35 of 47

False!

You may want to start collecting data before the product has launched! This would allow for a more rigorous evaluation.

You should begin collecting data right after the product launches.

36 of 47

Check the Early Data

Early in the process, it is important to check that the data is being collected as envisioned.

​

This is your opportunity to catch any errors, tweak the code, and verify that all the data is being stored properly. If possible, pull the data as you are planning to evaluate it, just to make sure it comes out how you are hoping.

​

You can also use this time to check for the following:

This may happen before the product has launched

  • Any harm being caused unintentionally due to the new product
  • Any immediate issues that are coming up for users that could improve experience quickly.

36

Better Government Lab

37 of 47

A True Story

A new portal for WIC (Special Supplemental Nutrition Program for Women, Infants, and Children) benefits launched and was being evaluated.

​

The team implemented a free response evaluation question to assess the administrative burdens.

​

By checking the early results, the team realized that program users were having a hard time finding out how to use their benefits to purchase diapers. The team was able to quickly improve the diaper-related information on the website. This easy fix was caught due to early data analysis.

37

Better Government Lab

38 of 47

Case Study Example - NY DOL

NY Department of Labor has launched the new product! After three weeks of running this new system, you have noticed that the number of Spanish-speaking applicants has decreased significantly. What would you do?

​

  1. Continue the roll-out
  2. Continue the roll-out and interview a case manager
  3. Stop the roll-out
  4. Pause the roll-out and conduct user interviews

38

Better Government Lab

39 of 47

Case Study Example - NY DOL

NY Department of Labor has launched the new product! After three weeks of running this new system, you have noticed that the number of Spanish-speaking applicants has decreased significantly. What would you do?

​

  • Continue the roll-out
  • Continue the roll-out and interview a case manager
  • Stop the roll-out
  • Pause the roll-out and conduct user interviews

39

Better Government Lab

While every case is different, it is important to consider how your product might be producing the opposite effect than intended. Pausing a roll-out and interviewing users will give more insight. Alternatively, this would be a good time to look through the data to see what could be going wrong.

40 of 47

Analyze the Data

Step 6

41 of 47

Time to Bring in the Experts

While we wish we could train every person on how to analyze this type of data, this slide deck would have to be at least 15 x the length… instead, we recommend you lean on the expertise either within your team, your organization, or outside experts to analyse the data.

​

Data analysis can include running regressions with different models.

​

The Better Government Lab is always a resource for inquiring about data analysis support for this type of project.

41

Better Government Lab

42 of 47

Share the Results

Step 7

43 of 47

Case Study - Congratulations! You Did It!

43

Better Government Lab

44 of 47

Summary - How to Design an Evaluation

Identify the outcomes of interest.

Step 1

Decide on a data collection strategy.

Step 2

Develop a timeline.

Step 3

Build data collection capacities alongside the product.

Step 4

Collect and check the data.

Step 5

Analyze the data.

Step 6

Share the results.

Step 7

44

Better Government Lab

45 of 47

Complimentary Materials

Program Evaluation

Administrative Burden Evaluation

Learn how program outcomes can be evaluated and some different strategies for evaluation. Review case studies to see these principles in action.

​

Become familiar with administrative burden and service outcome evaluation. Learn best practices, gain insight into survey questions, and review a case study to see this in action.

45

Better Government Lab

46 of 47

Contact the Better Government Lab

Excited to launch a civic technology evaluation?

47 of 47