1 of 17

Choosing a Program to Evaluate

Module 4

2 of 17

Introduction

2

3 of 17

By the end of this module, you will be able to...

Describe the parts of a successful evaluation plan,
Understand and apply criteria to choose potential programs, and
Identify an appropriate program to evaluate.

3

4 of 17

Describing the program
Identifying outcomes of interest
Articulating a theory of change
Creating a logic model
Matching outcomes of interest to indicators
Choosing methods to collect data
Creating a timeline and a budget

4

Parts of an Evaluation Plan

How can you ensure that your evaluation is successful? Start with a plan. The next seven modules in this workshop will take you through the steps of creating an evaluation plan.

The first step is identifying and describing the program to be evaluated -- which is the goal of this module.
The next step is identifying the outcomes of interest and the results you want to measure. Module 5 will help you determine what you want participants to learn from your program.
Next, you will articulate a theory of change, which is what we think is supposed to happen during and after a program. During Module 6, you’ll fill in the “missing middle,” the part between what you want to happen, and what actually gets done.
Then, you’ll create a logic model, which serves as a roadmap for your program. The focus of module 7 will help you to understand all the parts of this model.
Next, you’ll begin to match your outcomes of interest to indicators. Indicators provide evidence to let you know if you have reached the goals that you set. Module 8 will help you match your goals with indicators of success.
And lastly, in Module 9, you will choose how to collect data.
Once you’ve made a plan, then it’s time to create a timeline and a budget. Constant considerations are time and money!

5 of 17

Program Evaluation

5

(NSF, 2010)

For the purposes of ImpactNC, we want to evaluate one specific part of your organization’s work. Instead of trying to evaluate the overall success or outcomes of your museum or organization, or of every single one of your programs, you will select just one of the programs that your group implements for this evaluation project.

OLD TEXT:

According to the National Science Foundation, there are two basic levels of evaluation: program evaluation and project evaluation.

A program, not to be confused with a class, is a coordinated approach to exploring a specific topic area, made up of a collection of projects designed to meet certain goals. Examples might be the kids’ garden program, or the live animals program.

A project is a particular activity under the umbrella of your program. Examples might be a class on seeds for parents and preschoolers in the kid’s garden program or a nocturnal animals encounter for school groups within the live animals program. Project evaluation helps it develop and progress.

This work, ImpactNC is focused on project evaluation methods.

6 of 17

Choosing a Program to Evaluate

6

Outcomes of other organizations’ projects

Outcomes of your program

=

7 of 17

Questions to Consider When Choosing a Program

What are your evaluation priorities?

What resources are available?

What is the budget?

What is the timeframe for the evaluation program?

What outcomes do you want to measure?

7

So how do you begin to choose?

First, Think about your organization’s evaluation priorities.

What kind of evaluation will address the greatest need? What kind of evaluation will your organization support?

Then, What resources are available to you? Think about people, their availability, and their expertise.

Third, What types of evaluation fit within your organization’s budget?

Also, How long will this evaluation take? Ideally, it can be tackled during the same timeframe as ImpactNC so you can fully benefit from the support provided.

Lastly, and maybe most importantly, think about outcomes that are common to other organizations. These outcomes will probably be more general, like engagement or appreciation.

Before continuing, spend some time thinking about the criteria and the questions above. Then it’s time to narrow down your choice of program.

8 of 17

Questions to Consider Once You’ve Chosen a Program

What is the history of this program?�
What are the goals for this program? �
What are the types of activities that are involved? �
Who are all of the stakeholders that are involved in this program?

8

Now that you’ve chosen a program to evaluate, it’s time to consider its specific characteristics.

First, What’s the history of the program? Has it existed at your organization for a long time or is it relatively new? Has it gone through a lot of changes during its life?

Gathering documents like program educational materials, grant proposals, and any prior evaluation results helps show a program’s history.

Second, What are the goals of the program? What do you expect people to get from it? What would it look like for the program to be successful? If you walked in while the program was going on, what would tell you that it’s reaching its goals? What are people doing? What are they saying? We will use these goals to help determine outcomes.

Third, What are the types of activities that are involved? What do people do when participating? What is the experience of the participants? How do people feel and what do they encounter when participating? The experiences of the participants will also help determine outcomes, especially ones that are more effective, like appreciation and interest.

Fourth, Who are all of the stakeholders that are involved in this program? Who are the audiences? Think about everyone who is invested in its success: this includes audiences, educators, the community, and your funders.

Again, pause and take some time to consider how the characteristics of the prospective program fit with the goals of evaluation.

On the module worksheet, begin developing the plan by describing the proposed program using the guiding questions above.

9 of 17

What type of evaluation is needed?

9

Front-End: audience needs/interests

Formative: working?/

improvements/

changes

Summative: outcomes?/

strengths/

weaknesses

Life of the program

(CAISE, 2011)

Now that you’re on your way to choosing a program, it’s time to think about what type of evaluation you need. Think back to module 1 and the three types of evaluation we talked about: front-end, formative, and summative. Each type gives different kinds of feedback.

In order to choose the best type of evaluation, you will need to be thinking about your program and what you want to accomplish. For the purposes of ImpactNC, you’ll want to focus only on one type that best addresses the needs, time, and budget of your chosen evaluation project. In the future, you may find it helpful to use evaluations throughout the life of the program, taking advantage of the strengths of each type.

Front end evaluation takes place before the program. It helps to provide information about what your intended audience is interested in, what they know, and what they want to know.

It can also give information about the context surrounding the program like cultural or political influences.

Formative evaluation gives input on a program while it is underway. It will help determine how the program is (or isn’t) working. The data collected can be used to improve or change the program

Summative evaluation is used to provide feedback on a program that has concluded. It is used to figure out if the program reached its intended outcomes and figure out its strengths and weaknesses.

Where in the life of the program are you? What do you want to know? Think about these questions to help you choose the type of evaluation you want to do.

10 of 17

Developing Evaluation Questions

What do you and other stakeholders want to know about this program?

Answers must be

evidence-based,
aligned with your chosen outcomes, and
relevant and meaningful to you and other stakeholders.

10

(CAISE, 2011)

Now it’s time to think about developing your evaluation questions. These are not questions you will ask your participants but, instead, questions that will help you focus on what you want to know at the end of the evaluation.

The questions you ask will depend on the type of evaluation you’ve chosen, the amount of time you have, and the tools and resources needed.

The answers to questions you ask must be based on evidence from data that you gather, things like attendance records, survey results, or interview responses. Answers can take many forms. It can be numerical data from surveys, photos from your observations, or quotes from talking with participants.

Answers should be aligned with your chosen outcomes, which you’ll learn more about in the next module.

And, finally, answers should be relevant and meaningful to you and other stakeholders.

11 of 17

Wrap up / Highlights

Choosing a program..
Identifying outcomes.
Articulating a theory of change.
Creating a logic model.
Matching outcomes to indicators.
Choosing methods.

11

12 of 17

Conclusion

12

13 of 17

In the next module, you will be able to…

Describe what a learning outcome is,
Become familiar with two learning outcomes frameworks, and
Identify unanticipated outcomes.

13

14 of 17

For further information

National Science Foundation. (2010). The 2010 User-Friendly Handbook for Project Evaluation. Retrieved from https://www.purdue.edu/research/docs/pdf/2010NSFuser-friendlyhandbookforprojectevaluation.pdf

14

15 of 17

Stay connected

Visit us: https://research.ced.ncsu.edu/impactnc/

Dr. K.C. Busch

Assistant Professor of STEM Education

North Carolina State University | College of Education

kbusch@ncsu.edu

15

16 of 17

Credits

K.C. Busch

Lynn Chesnut

Regina Ayala Chávez

Aimee Fraulo

Kathryn T. Stevenson

Katy May

Lincoln Larson

Madeline Hinckle

16