1 of 23

Welcome!

As you enter the webinar, please introduce yourself via the chat

by sharing with the group:

Your PSU & role

&

Your FAVORITE thing to do this time of the year

We thank you for your time and attendance

1

https://bit.ly/OLRdecember14

OFFICE OF LEARNING RECOVERY & ACCELERATION

2 of 23

USED Non-Regulatory Guidance:

Using Evidence to Strengthen Education Investments

Dr. Richard Lewis, ARP Program Administrator

Jay Johnson, Program Manager

Rachel Wright-Junio, Director

OFFICE OF LEARNING RECOVERY & ACCELERATION

3 of 23

OFFICE OF LEARNING RECOVERY & ACCELERATION

4 of 23

Purpose

Assist PSUs in making wise choices with their investments during the last year of ESSER funds to ensure the greatest positive impact on students

Outcomes

Ensure that all PSUs are aware of the new Non- Regulatory Guidance for Using Evidence to Strengthen Education Investments resource from USED
Provide an overview of the document
Provide an example of how PSUs may utilize this document within their own contexts

OLR District USED Guidance Example

4

https://bit.ly/OLRdecember14

OFFICE OF LEARNING RECOVERY & ACCELERATION

5 of 23

Overview

Guidance Document Link:

https://www2.ed.gov/fund/grant/about/discretionary/2023-non-regulatory-guidance-evidence.pdf

Purpose of USED guidance: Help organizations successfully choose and implement evidence-based project components that are designed to improve outcomes for learners

Project Components: Evidence-Based activities, strategies, and interventions

5

https://bit.ly/OLRdecember14

OFFICE OF LEARNING RECOVERY & ACCELERATION

JJ: At the top of this slide you will see the link to the Guidance Document. It is in PDF form, so you may need to copy and paste it into your browser. The purpose of the USED Guidance Document is to help organizations successfully choose and implement evidence-based project components that are designed to improve outcomes for learners. Throughout the slides and guidance document, you will see references to “Project Components,” which is the USED’s way to refer to any evidence-based activities, strategies, or interventions. We are going to spend a short amount of time going over how the document is laid out, but we will spend most of our time talking through the process using our example district.

Guidance Document PDF (RL): https://www2.ed.gov/fund/grant/about/discretionary/2023-non-regulatory-guidance-evidence.pdf

6 of 23

Steps of the Process

6

U.S. Department of Education. (2023). Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments. Pg.4

OFFICE OF LEARNING RECOVERY & ACCELERATION

7 of 23

7

What is involved in the specific step

Questions to consider during the step

U.S. Department of Education. (2023). Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments. Pg.5

OFFICE OF LEARNING RECOVERY & ACCELERATION

8 of 23

Part 1: Section 1: Identify Local Needs

What are the needs of the learner population being served?

Who should be at the table with you?

Families
Students
Educators
State and local governments
community partners/stakeholders

8

Do you reach all stakeholders, especially from underserved groups?

Are you ensuring diversity and inclusion?

How do we identify local needs? 🧐

OFFICE OF LEARNING RECOVERY & ACCELERATION

9 of 23

Part 1: Section 1: Identify Local Needs

What are the needs of the learner population being served?

Data sources you may already have:

OLR R&E

NCStar

CCIP

9

Remember to use the data you have.

What else (other data) can you use to help better understand your local learner population?

How will you determine what should be prioritized?

What do we have to inform us about our local needs? 🧐

OFFICE OF LEARNING RECOVERY & ACCELERATION

10 of 23

Part 1: Section 2:

Select Relevant, Evidence-Based Project Components

Explore the broadest possible range of evidence

WWC | Find What Works! (ed.gov)

Select Project Components that are supported by evidence and have been demonstrated to be effective
Select Project Components that are feasible to implement

10

How do we know if it is relevant to our needs? 🧐

OFFICE OF LEARNING RECOVERY & ACCELERATION

When you are starting the process of selecting successful project components, you will want to begin by exploring the broadest possible range of evidence. The best place to start, is the What Works Clearinghouse. The WWC reviews research, determines which studies meet rigorous standards, and summarizes findings.

WWC | Find What Works! (ed.gov)

In our case, we wanted to find an evidence-based intervention that focused on sixth grade math and we came across a study called “The Effects of Math Video Games on Learning.” We checked to ensure that the setting and population used in the study were similar to our school districts population, and there were no great differences. We also considered the questions: What is the capacity to implement this project component or larger intervention? “School and community buy-in.” Are there available funds? “Yes.” Do staff have the needed skills? “Professional Dev. will be included.” Is there an implementation partner, as needed, to help? “Yes, Center for Advanced Technology in Schools”

11 of 23

Additional Federal Evidence Clearinghouses

Dept. of Labor’s Clearinghouse for Labor Evaluation and Research

Homepage | CLEAR (dol.gov)

Department of Justice’s CrimeSolutions

Reliable Research. Real Results. | CrimeSolutions, National Institute of Justice (ojp.gov)

Dept. of Health and Human Services’ Evidence-Based Practices Resource Center

Resource Center | SAMHSA

Americorps’ Evidence Exchange

Evidence Exchange | AmeriCorps

11

Where would I go for some cool resources/ideas? 😎

OFFICE OF LEARNING RECOVERY & ACCELERATION

12 of 23

Considerations

The target population and setting in which the evidence used to support the project component was conducted

Ability to implement

Available funding, staff resources, staff skills, and support for sustaining a project component

Database of Federally Funded Technical Assistance and Research Centers | OSEP Ideas That Work

List of additional questions to consider when selecting evidence-based project components can be found on pages 7 & 8 of the guidance document

12

OFFICE OF LEARNING RECOVERY & ACCELERATION

Some considerations to think about when reviewing potential project components.

When evidence supporting a project component is aligned to an org’s setting, it may increase the likelihood that prior positive outcomes will be replicated. In practice, finding evidence that was built in a context exactly like the one in which it is going to be applied can be difficult. (This is one reason why the Department encourages evidence building whenever possible: it increases the amount of evidence that helps educators understand what “works,” for whom, and under what conditions.) Project components supported by evidence for a specific setting and/or population that are not the same population and/or setting of an organization might meet a lower evidence level for the organization’s setting and/or population. We will discuss these levels later on. When relevant evidence is limited, organizations may seek to innovate by leveraging project components that are based on positive findings from high-quality research and evaluation activities conducted in contexts, settings, or populations that differ from their own. In those cases, those project components—when paired with a logic model that demonstrates how the project component is meant to improve an important learner outcome and with a plan to build high-quality evidence— might meet a less rigorous tier of evidence.

Organizations should also consider their ability to implement a given project component before deciding on an intervention or program. Organizations can work with technical assistance providers, including those supported by the Department of Education, to improve their capacity to implement evidence-based project components. A list of some Department-funded providers can be found at the link included in this slide: https://osepideasthatwork.org/find-center-or-grant/find-a-center

In the guidance document, on pages 7 and 8, there is a list of additional questions to consider when selecting project components and I would highly recommend using that as a checklist for reviewing potential interventions.

13 of 23

Part 1: Section 3: Plan for Implementation

Create a Logic Model (OLR Logic Model - v3.0)

SMART goals

Specific, Measurable, Achievable, Relevant, and Time-Bound

Clearly outlined roles and responsibilities for people and organizations involved

Timeline for implementation

List of resources required

Strategies to monitor and evaluate program effectiveness

13

What is our plan for implementation? 🧐

OFFICE OF LEARNING RECOVERY & ACCELERATION

14 of 23

Part 1: Section 3: Plan for Implementation

Basic Logic Model Template

14

Resources

Inputs

Activities

Outputs

Outcomes

Impact

Planned work

Your intended results

From: W.K. Kellogg Foundation (2004). Logic Model Development Guide. Retrieved from https://wkkf.issuelab.org/resource/logic-model-development-guide.html

1

2

3

4

5

Certain resources are needed to operate your program

If you have access to them, then you can use them to accomplish your planned activities

If you accomplish your planned activities, then you will hopefully deliver the service that you intended

If these benefits to participants are achieved, then certain changes in organizations, communities, or systems might be expected to occur

If you accomplish your planned activities to the extent you intended, then your participants will benefit in certain ways

OFFICE OF LEARNING RECOVERY & ACCELERATION

15 of 23

Part 1: Section 3: Plan for Implementation

SMART (specific, measurable, achievable, relevant, and time-bound) Goals:

Short Term Goals (this school year)
Long-term goal (next 3 years)

Roles and Responsibilities:

CATS, Teachers, Administrative Staff

Implementation timelines:

Professional Development - October
Classrooms will begin using the fractions-focus video games - November.

15

What is our plan for implementation? 🧐

Post-Program Assessments - no more than 10 days after the conclusion of the program
Feedback - after the first “round” of implementation and post-program assessment results

Resources required to support the project component:

Funding to contract with CATS for the PD and software
Buy-in from the participating teachers
Time to allow teachers to participate in the required Professional Development

OFFICE OF LEARNING RECOVERY & ACCELERATION

For our sample district, we included a basic, sample logic model based on the previous slide within the example doc (but you are aslo welcome to check out our office’s much more in depth logic model linked previously), We also crafted both short and long term SMART Goals that we wanted to achieve. In the short term (this school year), we want to see 60% of students score an 80% or above on the post-program assessment, 90% of students report having a better understanding of fractions than they did before participating in the program and 90% of teachers report that their students show an increase in their understanding of fractions after participating in the program. Our long-term goal is to see a 5% or greater improvement in 6th-grade Math EOG scores in the next 3 years.

We defined the roles and responsibilities of the contracted provider, teachers, and administrators, developed a specific timeline for implementation, and listed the resources necessary to support the project.

And I’ll now pass the mic back to Richard to talk through implementation of a project component once a plan is in place.

16 of 23

Part 1: Section 4: Implement

How will you collect relevant information about how your efforts are working and how to make changes?

What data do you have?
Is this occurring with fidelity?
Resource usage
Who’s at the table?
Scalability

16

Is the implementation plan being followed, including partner roles in implementation? If not, why not?

What are unforeseen barriers or potential biases in assumptions or inaccuracies to successful implementation?

What is our plan to ensure fidelity?

How will we collect data?

🧐

OFFICE OF LEARNING RECOVERY & ACCELERATION

17 of 23

Part 1: Section 5: Examine & Reflect

Performance Monitoring - involves regularly collecting and analyzing data to track progress against targets and goals, including data on program outputs and when available, program outcomes.

Logic Model expectations

Interim goals/milestones
continuous improvement insights

17

What are the expectations of success and how can success be measured?

What are interim progress and performance milestones that can be tracked?

What have participants (i.e., learners and educators) shared about their experience?

How did we define progress?

How do we define success?

🧐

OFFICE OF LEARNING RECOVERY & ACCELERATION

18 of 23

Part 1: Section 5: Examine & Reflect

Performance Evaluation - demonstrates whether a project component (or the project as a whole) is causing an observed outcome.

Evidence of effectiveness

Impact of project on relevant outcomes

Continue
Modify
Discontinue

How will you share with others?

18

What forms of evidence—performance monitoring, evaluation, or both—are needed for decision-making?

Are evidence needs expected to change over time?

Did we meet our goals?

Where do we go from here?

🧐

OFFICE OF LEARNING RECOVERY & ACCELERATION

19 of 23

Part 2: Guidance on the Meaning of “Evidence-Based”

Supported by strong evidence, moderate evidence, promising evidence, or evidence that use of the component demonstrates a rationale.

19

U.S. Department of Education. (2023). Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments. Pg.12

OFFICE OF LEARNING RECOVERY & ACCELERATION

The Every Student Succeeds Act defines an evidence-based project component as being supported by strong evidence, moderate evidence, promising evidence, or evidence that the use of the component demonstrates a rationale. Some ESEA programs encourage the use of evidence-based project components while others require it, so if you are applying for a federal grant in the future, please use the program-specific guidance to determine if a specific level of evidence is required for your project components. The levels of evidence are defined in the image from the guidance document included here.

A project with the highest level of evidence is based on strong evidence from at least one well designed and well implemented experimental study,
A project supported by the second tier of evidence is based on moderate evidence from at least one well designed and well implemented quasi experimental study, and
A project is supported by the third tier of evidence if it is based on promising evidence from at least one well designed and well implemented correlational study with statistical controls for selection bias.
A project can also be supported by evidence if it “demonstrates a rationale based on high quality research findings or positive evaluation that such activity, strategy, or intervention is likely to improve student outcomes or other relevant outcomes and includes ongoing efforts to examine the effects of such activity, strategy, or intervention.”

20 of 23

20

Evidence Required	Level of Evidence
Evidence Required	Strong Evidence	Moderate Evidence	Promising Evidence	Demonstrate a Rationale
Outcomes	At least one statistically significant and positive effect on a relevant outcome; no statistically significant and negative effects on a relevant outcome	At least one statistically significant and positive effect on a relevant outcome; no statistically significant and negative effects on a relevant outcome	At least one statistically significant and positive effect on a relevant outcome	N/A
Study Design	Experimental Study	Experimental study or quasi-experimental design study	Experimental study, quasi-experimental design study, or correlational study with statistical controls for selection bias	Logic model informed by research or evaluation findings
WWC Evidence Rating	Meets WWC without reservation	Meets WWC with or without reservation	N/A	N/A
Sample Size	A large sample (n=350+) and a multi-site sample	A large sample (n=350+) and a mult-site sample	N/A	N/A

U.S. Department of Education. (2023). Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments. Pg.16

OFFICE OF LEARNING RECOVERY & ACCELERATION

21 of 23

Sign Up for OLR’s Listserv

Navigate to bit.ly/GovDeliveryDPI

Select OLR from the “Internal Lists” category

OR

Request to be added manually by sending an email with your request to richard.lewis@dpi.nc.gov

21

OFFICE OF LEARNING RECOVERY & ACCELERATION

22 of 23

Dr. Richard W. Lewis, richard.lewis@dpi.nc.gov

ARP Program Administrator

Office of Learning Recovery and Acceleration

Jay Johnson, jay.johnson@dpi.nc.gov

Program Manager

Office of Learning Recovery and Acceleration

Rachel Wright Junio, rachel.wrightjunio@dpi.nc.gov

Director

Office of Learning Recovery and Acceleration

OLR Program Team

OFFICE OF LEARNING RECOVERY & ACCELERATION

23 of 23

Thank you

Please complete our

exit ticket

https://bit.ly/OLRdecember14

OFFICE OF LEARNING RECOVERY & ACCELERATION