Investing in What Works 101

$2 trillion opportunity

Each year local, state, and federal governments in the United States spend more than $2 trillion to meet the needs of our most vulnerable populations.

​

These investments can produce positive economic mobility outcomes for all if these governments partner with communities to determine priorities, set a shared vision, prioritize services with evidence of effectiveness and ensure continuous improvement.

We know a great deal about what works

Federally- and philanthropically-funded research clearinghouses identify more than 2,000 interventions with evidence of effectiveness.

Results for America’s Economic Mobility Catalog helps government leaders identify evidence-based interventions to address economic mobility-related challenges. The catalog contains interventions and case studies in the following areas:

• K-12 education
• Workforce development and post-secondary
• Early childhood
• Other economic mobility area

Through EdResearch for Action, Results for America and the Annenberg Institute at Brown University have released more than 20 briefs to help educators implement evidence-based interventions.

3 ways to invest in what works

Government leaders can drive positive population outcomes for all by shifting funding toward evidence-based interventions through:

• Grantmaking and contracting
• Budgeting
• Delivery of direct services by government employees

Grantmaking and Contracting�

Governments increasingly invest in what works through grants and contracts

Governments increasingly define and prioritize evidence of effectiveness in grant programs in order to produce better outcomes for all.

In 2023, eleven federal departments/agencies invested $30 billion in programs that define, prioritize and/or encourage evidence of effectiveness in order to produce better outcomes in education, workforce development, health, human services, housing, and public safety

Through 2023, 34 states invested $1.6 billion in 79 such programs in 56 different state agencies

A number of local governments have done similarly (Results for America is now researching how widespread this practice is in local government)

Step 1: Define evidence of effectiveness in your request for grant/contract proposals

​

Definitions of evidence of effectiveness allow policymakers and those who deliver services to have a shared understanding of which interventions have qualifying evidence.         

You can use your own definition of evidence of effectiveness, if one already exists, or you can adopt or adapt a framework from elsewhere, such as:

• Education (Every Student Succeeds Act, What Works Clearinghouse)
• Workforce (Clearinghouse for Labor Evaluation and Research, Pennsylvania, Texas)
• Other federal (American Rescue Plan, Americorps)
• Other state (Colorado, Minnesota, Tennessee)

Strategy A: Require that funding be used to support evidence-based interventions

• Education (federal): ESSA Title I, Part A School Improvement 7% Set-Aside
• Education (state): Tennessee High-Dosage Tutoring
• Workforce (federal): Reemployment Services and Eligibility Assessment (RESEA)
• Early childhood (federal): Home Visiting
• Early childhood (local): Sonoma County AVANCE Parent-Child Education

Strategy B: Allocate points to grant applications that meet the government’s evidence definitions

• Education (federal): Comprehensive Literacy State Development Program
• Education (state): Nevada Adult Education and Family Literacy
• Workforce (federal): Reentry Employment Opportunities
• Workforce (state): Texas Building Construction Trades

Step 2: Select an evidence prioritization strategy for your request for grant/contract proposals

When designing your prioritization strategy, you’ll want to consider the:

• Extent to which existing evidence-based interventions can achieve your program’s goals
• Importance placed on proven interventions versus innovation and evidence building
• Degree of uniformity desired in service availability across the communities that will be served
• Capacity of potential applicants to deliver various interventions

Considerations for prioritization strategies

98 federal grant programs define and prioritize evidence effectiveness, including:

• 18 in education
• 13 in postsecondary education and workforce development
• 4 in early childhood
• 63 in other economic mobility categories

​

Summary of Federal Grant Programs that Define and Prioritize Evidence

79 state grant programs define and prioritize evidence effectiveness, including:

• 27 in education
• 8 in workforce development
• 4 in early childhood
• 40 in other economic mobility categories

​

​

Summary of State Grant Programs that Define and Prioritize Evidence

The U.S. Department of Education's Comprehensive Literacy State Development Program ($188 million in FY22) helps states advance student literacy skills.

The program’s solicitation for grant applications:

• Defines evidence of effectiveness as interventions that demonstrates a statistically significant effect on improving student outcomes or other relevant outcomes based on:
• Strong evidence from at least one well-designed and implemented experimental design study
• Moderate evidence from at least one well-designed and implemented quasi-experimental design study
• Promising evidence from at least one well-designed and implemented correlational design study with statistical controls for selection bias.
• Prioritizes evidence of effectiveness by awarding up to 25 points based on quality of project design, which includes the extent to which funding will be used for evidence-based activities.

​

​

Federal Defining and Prioritizing Evidence in Education Example

The Tennessee Department of Education's High-Dosage Tutoring program ($27 million in 2021) helps advance student literacy skills.

The program’s solicitation for grant applications:

• Defines evidence of effectiveness as high-dosage low-ratio tutoring with the following features:
• students receive tutoring at least 2-3 times per week
• 30-45 minute sessions
• ratios of no more than 1:3 (grades 1-5) or 1:4 (grades 6-8)
• Prioritizes evidence of effectiveness by exclusively funding evidence-based high-dosage low-ratio tutoring for students

​

​

State Defining and Prioritizing Evidence in Education Example

The Texas Workforce Commission's Building Construction Trades program ($1.8 million in 2023) prepares youth for employment in construction occupations.

The program’s solicitation for grant applications:

• Defines evidence of effectiveness through an evidence-based grantmaking framework developed by the commission
• Prioritizes evidence of effectiveness by awarding up to 10 points based on the level of causal evidence

Texas initially defined and prioritize evidence for this program in 2021 and is using a similar approach in other workforce and higher education programs.

​

State Defining and Prioritizing Evidence in Workforce Example

The U.S. Department of Health and Human Services’ Maternal, Infant, and Early Childhood Home Visiting program ($409 million in FY22) seeks to produce positive maternal and child health outcomes.

The program’s solicitation for grant applications:

• Defines evidence of effectiveness as having at least:
• one qualifying impact evaluation finding favorable statistically significant impacts in two or more outcome domains and/or
• two qualifying impact evaluations finding favorable statistically significant impacts in one outcome domain
• Prioritizes evidence of effectiveness through a requirement that 75% of funds support qualifying models

Federal Defining and Prioritizing Evidence in Early Childhood Example

The Michigan Department of Education's Great Start Readiness Program program ($338 million in 2022) supports kindergarten readiness.

The program’s solicitation for grant applications:

• Defines evidence of effectiveness through key features of this preschool intervention that have been found to produce positive effects for participating children through multiple quasi-experimental design impact evaluations and are outlined in program quality standards.
• Prioritizes evidence of effectiveness by requiring applicants to use the Great Start Readiness preschool education model.

​

​

State Defining and Prioritizing Evidence in Early Childhood Example

Budgeting�

Using evidence in the budget process

​

The federal government currently does not have a budget process that strongly encourages or requires federal agencies to define and prioritize evidence in specific grant or direct service programs.

Four states (CO, MN, NM, TN) currently have budget processes that encourage agencies to define and prioritize evidence in specific grant or direct service programs.

RFA is currently reviewing whether any local governments have a budget process/system that strongly encourages or requires agencies to define and prioritize evidence in their grant-making/contracting or their direct services.

​

Step 1: Define evidence of effectiveness for use in the budget process

Definitions of evidence of effectiveness allow policymakers and those who deliver services to have a shared understanding of which interventions have qualifying evidence.         

You can use your definition of evidence of effectiveness, if one already exists or you can adopt or adapt a framework from elsewhere, such as:

Minnesota (6 categories)

• Proven Effective
• Promising
• Theory based
• Mixed effects
• No effect
• Proven Harmful

Colorado (4 categories)

• Proven
• Evidence-informed
• Theory-informed
• Opinion-based

New Mexico (3 categories)

• Evidence-based
• Research-based
• Promising

Tennessee (5 categories)

• Strong Evidence
• Evidence
• Outcomes
• Outputs
• Logic model

Step 2: Request evidence in budget proposals

​

Example from

State of Minnesota

• When sending out budget guidance to agencies, the chief executive and central budget office request evidence of effectiveness for any proposals.
• This prompts agency leaders and the budget staff who prepare these requests to consider evidence of effectiveness when developing their proposals and provides documentation that can be reviewed by policy advisors and central budget staff in preparation for decision-making.

Step 3: Support development of evidence-based budget requests

​

Example from

State of Minnesota

Central budget or evidence teams can support agencies in developing evidence-based proposals through structured training, such as the brief virtual modules pictured below that were developed in 2018 by the State of Minnesota, and through tailored coaching and assistance.

Step 4: Prioritize evidence using decision materials that highlight evidence-based items

Example from

State of Minnesota

Minnesota

​

Since 2016 Minnesota:

• Defined tiers of evidence (or lack thereof) that are used to determine whether a given budget proposal will support interventions with evidence or effectiveness and/or an impact evaluation to produce evidence of effectiveness, so that it can be prioritized.
• Developed a program inventory that transparently reports the evidence category for 733 publicly funded interventions
• Offered trainings to agency staff who prepare budget proposals

For FY 2024-2025:

• The Governor and central budget office have directed agencies to include information about the evidence base for their budget proposals, a summary of evidence, citations, and amount to be spent on the activity.
• The central budget office has reviewed proposals for evidence of effectiveness and prepared a prioritized list of those proposals with qualifying evidence, for use by the Governor in developing the budget

Colorado

​

Since 2016 Colorado:

• Defined tiers of evidence (established in 2016 and codified in 2021) that are used to determine whether a given budget proposal will support interventions with evidence or effectiveness and/or an impact evaluation to produce evidence of effectiveness, so that it can be prioritized.
• Offered training (video and slides) to agency staff who prepare budget proposals

For FY 2022-2023:

• The Governor and central budget office have directed agencies to include information about the evidence base for their budget proposals, a summary of evidence, citations, and amount to be spent on the activity.
• The central budget office has reviewed proposals for evidence of effectiveness and prepared a prioritized list of those proposals with qualifying evidence, for use by the Governor in developing the budget

New Mexico

​

Since 2011 New Mexico:

• Defined tiers of evidence (codified in law in 2019) that are used to determine whether a given budget proposal will support interventions with evidence of effectiveness, so that it can be prioritized.
• For its budget process, required certain state agencies (selected annually by the state legislature) to “identify each sub-program as evidence-based, research-based, promising, or lacking evidence of effectiveness” and report on the amount allocated for each of these evidence tiers.

For FY 2023:

• The Legislative Finance Committee issued budget instructions that provide guidance for agencies for budget expansions (limited to committee priorities) and evidence-based programs as promulgated by the LFC’s Legislating for Results framework.
• Committee staff reviewed proposals for evidence of effectiveness and prepared a prioritized list of those proposals with qualifying evidence for use by the committee when making decisions.

Tennessee

​

Beginning in 2018, Tennessee has:

• Issued budget instructions to encourage state agencies to invest in programs that are supported by research and evidence.
• Invited agencies to, within their budget requests, highlight their programs’ level of evidence based on the five evidence steps defined by the state’s Office of Evidence and Impact, when completing cost increase and base reduction support forms.
• Given the Governor’s Office a summary report for use in the budget process that objectively presents the evidentiary information and any available data for each programmatic budget request to inform the governor’s decisions regarding those requests.

Direct Services Provided by Government Employees�

Direct Services

​

Local governments implement evidence-based practices with government employees, such as:

• Summer learning in Tuscaloosa, Alabama
• Summer Youth Employment in Louisville, Kentucky
• Nurse Family Partnership Home Visiting in Birmingham, Alabama

​

State governments also implement evidence-based practices with government employees, such as:

• Drug Courts in New Mexico
• Prisoner-Reentry Employment in Minnesota

RFA is currently working to identify additional federal, state, and local governments that directly administer evidence-based services.

Step 1: Define evidence of effectiveness

​

• Definitions of evidence of effectiveness allow policymakers and those who deliver services to have a shared understanding of which interventions have qualifying evidence.         
• You can use your definition of evidence of effectiveness, if one already exists or you can adopt or adapt a framework from elsewhere, such as:
• Education (Every Student Succeeds Act, What Works Clearinghouse)
• Workforce (Clearinghouse for Labor Evaluation and Research, Pennsylvania, Texas)
• Other federal (American Rescue Plan, Americorps)
• Other state and local (Colorado, Minnesota, Tennessee, Sonoma County)

Identify an evidence-based intervention that will be responsive to your jurisdiction’s most pressing needs and context, including:

• Community priorities
• Existing resources and capacity
• Prioritizing most vulnerable populations
• Alignment with other priorities and commitments

The Economic Mobility Catalog and other clearinghouses can be used to identify interventions with evidence of effectiveness.

Once selected, you can build support for implementation, integrate the intervention into policies and procedures, train staff across the organization, implement in a manner that ensures services are culturally competent, and conduct ongoing monitoring and evaluation.

Step 2: Prioritize evidence-based services

Next Steps

Government leaders can identify an area to start, such as:

• An upcoming grant solicitation
• An upcoming budget cycle
• A service delivered by your agency’s staff

Government leaders can invite Results for America to:

• Present to your team
• Help you improve your:
• requests for proposals
• budget guidance or proposals
• direct service plans

Next Steps