Introduction to Administrative Burden Evaluation
A Better Government Lab + Nava PBC Collaboration
Eleanor Grudin
Nava + Better Government Lab Graduate Research Fellow
Martelle Esposito, M.S., M.P.H.
Nava PBC
Eric Gianella, Ph.D.
Georgetown University
Michael Chen
Nava PBC
These slides were developed in a collaboration between the Better Government Lab and Nava PBC. The intention was to support training Nava staff on the importance of evaluation. We hope they are helpful to others in the Civic Tech community interested in learning about applied program evaluation.
ACKNOWLEDGEMENT
Contents
4
Better Government Lab
Defining “Administrative Burden” and “Service Outcomes”
SECTION 1
Reflection Question
What words come to mind when you think of applying for government services?
6
Better Government Lab
Reflection Question
What words come to mind when you think of applying for government services?
Many people associate negative emotions with government services - this can be caused by administrative burdens.
7
Better Government Lab
What are “Administrative Burdens”
Administrative burdens are at the core of the problem of most user experiences. They are the barriers to access for eligible program recipients, the deterrents to seeking support, and the source of much disillusionment with democratic bureaucracies.
8
Better Government Lab
Types of Administrative Burdens
Learning Costs
The challenges people face when trying to discover information about a program, such as program existence, eligibility, and requirements.
Psychological Costs
The negative feelings of stress, loss of autonomy, or stigma associated with the use of a program, which can materialize in a mental or emotional reaction to a barrier or design of the program.
Compliance Costs
The barriers for citizens to comply with rules and requirements of a program, whether it is to first participate in a program or maintain benefits over time.
9
Better Government Lab
Source: Herd, P. and Moynihan, D.P. (2019). Administrative burden: Policymaking by other means (1st ed.). Russell Sage Foundation.
Matching Activity
10
Better Government Lab
Having to research the 50 different housing assistance programs available in Washington, D.C..
Feeling ashamed because you have to put back an item your child wanted but was not covered by SNAP.
Forgetting to submit proof of job applications for unemployment benefits.
Being required to mail in a printed and signed application for WIC benefits.
Learning Costs
Psychological Costs
Compliance Costs
Matching Activity - Answers
11
Better Government Lab
Having to research the 50 different housing assistance programs available in Washington, D.C..
Being required to mail in a printed and signed application for WIC benefits.
Forgetting to submit proof of job applications for unemployment benefits.
Feeling ashamed because you have to put back an item your child wanted but was not covered by SNAP.
Learning Costs
Psychological Costs
Compliance Costs
What is a “Service Outcome”
Service outcomes are outcomes that measure the experience of service delivery.
12
Better Government Lab
Where Service Outcomes Fit In
Product Output
A working product has been created.
Ex: A new Veterans’ Affairs benefit portal launches.
Service Outcome
Because of the product, the end-user experience is improved.
Ex: Applicant frustration decreases due to the new SNAP application process.
Program Outcome
The product results in a directional change in the overarching outcomes of a group.
Ex: Adding a multilingual chatbot significantly decreases the number of rejections for WIC benefits in Spanish-speaking mothers.
13
Better Government Lab
Activity: Which is a Service Outcome?
Outcome 1
Outcome 2
Outcome 3
Outcome 4
Average time to complete a Medicaid application changes from 1 hour to 45 minutes.
A new chatbot is added to the SNAP application page.
There is an increase in non-English speaking families enrolled in SNAP.
The customer satisfaction increases when the number of user-reported errors within the FEMA aid portal decreases.
14
Better Government Lab
Which is a Service Outcome? - Answers
Service Outcome (Timeliness)
Product Output
Program Outcome
Service Outcome (Reliability)
Average time to complete a Medicaid application changes from 1 hour to 45 minutes.
A new chatbot is added to the SNAP application page.
There is an increase in non-English speaking families enrolled in SNAP.
The customer satisfaction increases when the number of user-reported errors within the FEMA aid portal decreases.
15
Better Government Lab
Evaluation Design and Why We Evaluate
SECTION 2
Why do we care about service� experience evaluations?
What works
We want to know what works and for whom.
Sharing information
Evaluation results can be shared with others in the field.
Empowering designers and product managers
Data empowers people who already knows what works but cannot always convince decision makers.
Quantifying the difference
Customers and funders can see that what you did made a difference.
17
Better Government Lab
What is a Service Evaluation in the Context of Product Development?
A systematic process of collecting and analyzing data to determine if the technology or service is achieving its service outcomes.
Causation vs. Correlation
18
Better Government Lab
Overview of Rigor Levels in Evaluation
19
Better Government Lab
Data collected before & after with Intervention + control groups and randomization
Data collected before & after with intervention + comparison groups
Data collected before & after
Data only collected after
Less rigorous and descriptive
(non-experimental)
More rigorous and causal
(experimental)
Level 1: Data Only Collected After
Sometimes, the best we can do is collect data after we have launched a product.While this will not be enough to prove causation, there are many stories that can be told with this type of data.
Ex: The DMV launches a new ID renewal portal and it features a new 5-star approval feature when someone submits their form. The data is only collected after the new portal is launched.
20
Better Government Lab
Level 2: Data Collected Before & After
A before and after picture is useful in seeing how things have changed. This is a way to find correlation. However, it is not enough to prove causation. Other world factors could be impacting the results.
Ex: 5-point satisfaction scores collected before the new product launch and after.
21
Better Government Lab
Level 3: Data Collected Before & After with Intervention & Comparison Group
This is beginning to get into rigorous study design. These types of evaluations are often called quasi-experimental. While they are still leaving the door open for endogenous variables, they generate very useful data that can be published! This is where gradual roll-out evaluations often fall.
Ex: Comparing the satisfaction scores of those who receive the intake form in the gradual roll-out to those who did not receive the new intake form.
22
Better Government Lab
Level 4: Data Collected Before & After with Intervention & Comparison Group & Randomization
This is a Randomized Control Trial (RCT). This is the most rigorous form of study design; however, it can be one of the most challenging to execute. This is the gold standard for academic publications.
Ex: Gradually rolling out a new renewal system for Medicaid benefits, where those who’s application ID ends in a 0,2,4,6,8 receive the new system and application IDs ending in a 1,3,5,7,9 do not.
23
Better Government Lab
The Power of Gradual Roll-Outs
Can you randomize who is in each phase of the roll-out?
Can you track who was in each phase?
If you have a gradual roll-out as part of the product implementation, you get a rigorous study design for free!
If so, you can perform a Randomized Control Trial (RCT) in which you compare the impact of the rolled-out product vs. the status quo.
By tracking who was in each phase of the roll-out, a comparison study can be performed (even if it is not random!) This study will be able to look at how the new product influences the program outcome.
24
Better Government Lab
Evaluating Administrative Burdens
SECTION 3
What tools can we use to enhance rigor in service experience evaluations?
26
Better Government Lab
It is easy to come up with bad survey questions…
27
Better Government Lab
How good did you feel when uploading your ID or completing this application?
A. Great! B. Good C. Neutral D. Bad E. Very Bad
Well, I didn’t like the uploading process, but I did like the application… so I’ll say neutral.
I didn’t care about the ID part but I HATED the application… so I’ll say very bad.
But it is hard to develop validated survey questions.
28
Better Government Lab
Using non-validated questions is like trying to compare apples to oranges… it just doesn’t quite work.
Options for Measuring Administrative Burden
1-Question Survey
3-Question Survey
Free Response Question
29
Better Government Lab
1-Question Administrative Burden Survey
30
Better Government Lab
Question | Answer |
Please think about your most recent experience with the program when you respond to the question. How would you describe this experience overall? | Scale: 1 - Very difficult 2 - Somewhat difficult 3 - Neither difficult nor easy 4 - Somewhat easy 5 - Very easy |
Highlighted in pink: This can be tailored to your specific needs.
Highlighted in yellow: This part should be used verbatim as it has been tested for validity.
3-Question Administrative Burden Survey
31
Better Government Lab
Highlighted in pink: This can be tailored to your specific needs.
Highlighted in yellow: This part should be used verbatim as it has been tested for validity.
Question | Answer |
Learning Costs How easy or difficult was the process of finding information about the program, such as how to apply or what you needed to do to renew your benefit? | Scale: 1 - Very difficult 2 - Somewhat difficult 3 - Neither difficult nor easy 4 - Somewhat easy 5 - Very easy |
Compliance Costs How was the process of filling out the paperwork, providing proof of eligibility (such as pay stubs, proof of residence, birth certificates, etc.), and/or attending interviews? | |
Psychological Costs Please describe how you felt during these experiences. | Scale [FRUSTRATED]: 1 – Extremely, 2 – Very, 3 - Moderately, 4 – Slightly, 5 – Not at all |
Free Response Question(s)
32
Better Government Lab
“What could we do to improve the application experience? Please be as specific as possible.”
To analyze these results, we recommend creating the following categories for tagging.
Stage of User Journey | Burden | General |
pre-application, intake application, ID Verification, post-submittal, certification and adjudication. | Learning, Compliance, or Psychological Cost | Positive or Negative Experience |
When to Use Each Survey Option
1-Question Survey
3-Question Survey
Free Response Question
33
Better Government Lab
Case Study
SECTION 3
Background of the Case
GetCalFresh Survey
35
Better Government Lab
Two Versions of the Survey
36
Better Government Lab
Single item scale
Three item scale
Findings from the Survey
37
Better Government Lab
Applicants who are uncertain about their determination outcome experience administrative burdens at similar levels to those who were denied.
Importantly, this relationship between uncertainty and burden levels persists regardless of the applicant's final determination status.
FINDING 1
Findings from the Survey
38
Better Government Lab
Applicants experiencing uncertainty were significantly less likely to provide positive feedback, while reporting higher rates of anxiety and frustration in their open-ended responses to the enrollment survey.
This suggests that uncertainty not only leads to higher levels of reported burdens, but also comes with psychological distress.
FINDING 2
Takeaways from this Case
39
Better Government Lab
A Real-World Example
SECTION 5
Application from 2025
41
Better Government Lab
Imagine the following survey…
42
Better Government Lab
Imagine the following survey…
43
Better Government Lab
Reflection Questions:
Final Survey Best Practices
SECTION 6
A short, clear, invitation that expresses the “why”.
Indicate how long the survey will take.
Text them the link for longer surveys so they can take it on the road.
Survey them as soon as possible after completing a task (ex: when they click submit)
Ex: Please complete this 3-minute survey to help us improve future users’ experiences.
45
Better Government Lab
Survey Best Practices
Conclusion
Program outcomes represent real-world differences.
Unlike building a product or improving user experience, program outcomes focus on the tangible, directional changes in a group's overarching goals.
Rigorous program outcome evaluation is possible.
While different levels of rigor exist, even a gradual rollout of a new system can provide a powerful opportunity to conduct a rigorous study that compares the new product to the status quo.
Evaluation proves what works.
By systematically collecting and analyzing data, a program evaluation can demonstrate if a technology or service is actually achieving its intended outcomes. This provides crucial evidence for stakeholders and funders
A rigorous evaluation can lead to widespread impact.
A successful program evaluation, like the case study on SNAP interviews, can lead to the widespread adoption of effective solutions by other organizations and states, creating a ripple effect of positive change.
46
Better Government Lab
Complimentary Materials
Program Evaluation
Designing an Evaluation
Learn how program outcomes can be evaluated and some different strategies for evaluation. Review case studies to see these principles in action.
Learn the step-by-step process for designing and implementing an evaluation in the civic technology context. Practice your new knowledge on an in-depth case study.
47
Better Government Lab
Discussion Questions
SECTION 5
Discussion Questions
49
Better Government Lab