1 of 20

Culturally Competent Evaluation

Module 3

1

2 of 20

Introduction

2

3 of 20

By the end of this module, you will be able to...

Become familiar with what culture and cultural competence is,
Define where cultural competence is found in evaluation,
Learn the difference between diversity and equity, and
Identify ethics in culturally responsive evaluations.

3

4 of 20

What is culture?

The shared experiences of people
Race/ethnicity, religion, social class, sexual �orientation, gender, age, and ableness
Culture is dynamic
Culture shapes experience

4

5 of 20

What is cultural competence?

Ability to understand, communicate with, and effectively interact with people across cultures.

Awareness of one's own cultural position and others’ positions.

Use methods that match the context and the diverse values and perspectives of people.

5

6 of 20

Why is cultural competence important?

Ethically necessary
Improves validity
Theoretically sound

6

(AEA, 2011)

Cultural competence is important for three different reasons.

First - Cultural competence is an ethical issue. It represents the effort of the evaluator to produce honest and valid work that respects different cultures. An example of this might be using the participants’ own language for the consent process, rather than English.

Second - Cultural competence is also a validity issue. Cultural competence helps minimize any bias that stereotypes or differing cultures might cause. It helps build trust and open communication between participants and the evaluator. When interpreting results, you should do so in ways that are relevant and respectful to your participants.

And Third - The theories we use are not objective or neutral. They come from our understanding of culture and they are shaped by our own cultural values. Not all the theories are going to fit within the cultural context of our work, so we need to be careful to choose an appropriate theory that is sensitive to different cultures.

7 of 20

Essential mind-set for cultural competence

Evaluation is not culture free.
Cultural identity is complex.
Power dynamics influence culture.
Language is a powerful indicator of culture.

7

(AEA, 2011)

There is a certain mindset that can help us to be more culturally competent.

First, We need to acknowledge that evaluation is not culturally neutral. the methods, tools for data collection, analysis, and representation of results are not culture-free. It’s important to think about how knowledge is built and communicated in different cultures. �
Second, it is important to acknowledge the complexity of cultural identity. People often identify with multiple cultural groups, so we have to be careful not to assume which groups someone belongs to.

Next, we need to recognize any power dynamics: �Different cultures can have different levels of status and power. �Some cultures might have privileges that they aren’t even aware of. So as evaluators, we need to be careful not to reinforce cultural stereotypes and prejudice in our work.

Last, Language is powerful, and a particular characteristic of each culture. We as evaluators need to use thoughtful language to recognize and reduce communicating bias.

�

8 of 20

When is cultural competence important?

8

Preparation

Engaging stakeholders

Design

Instrumentation

Collecting and analyzing data

Communicating results

(NSF, 2010)

Cultural competence is not just one action to be taken in an evaluation. We need to be culturally competent throughout the evaluation process.

From the beginning, when we as evaluators are preparing the evaluation, we should carefully work to understand the cultural and sociopolitical context of the program being evaluated. For example, we could involve stakeholders in the very beginning of the process and start a conversation about what is important to them.

In the middle of the evaluation process, we can be sensitive to culture when we design and implement the evaluation.

For example, when choosing or developing tools for use with a Spanish-speaking population, like a survey or questionnaire, we should develop these tools in Spanish with input from a member of the community.

When we finish our evaluation, and want to report our results, we need to make sure that we give an accurate, complete, and culturally respectful representation of our participants. For example, we can present results in a way that is understandable to our stakeholders.

9 of 20

“To ensure recognition, accurate interpretation, and RESPECT FOR DIVERSITY, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence.”

9

(AEA, 2011, p.1)

10 of 20

What is diversity?

10

11 of 20

What is inclusion and equity?

11

HAS EVERYONE’S VOICE BEEN WHO IS IN THE ROOM? WHO IS TRYING TO GET IN THE

HEARD IN THE ROOM? ROOM, BUT CAN’T?

12 of 20

How can culturally competent evaluation contribute to equity and diversity in ISE?

Privilege community voice
Identify desired outcomes
Multicultural validity
Broadening participation
Evaluate community-based partnerships
Replication and scaling

12

(Garibay & Teasdale, 2019)

There are six ways that culturally competent evaluation can advance equitable and inclusive practices:

Privileging the voices and experiences of all community members, of all different cultures, especially those who have been marginalized.
Evaluators should consider the community perspectives during the whole process and identify the desired outcomes.
Evaluators should make sure that instruments, measures, and results are valid and trustworthy.
As evaluators, when thinking about how we define “broadening participation”, we have to examine our own privilege and biases, and consider how these might affect our work.
Often partnering with local communities helps broaden participation.
Another technique to broaden participation is to carefully replicate programs that have already been successful and then expand them to reach more people and communities.

13 of 20

Ethics in culturally responsive evaluations

Procedural

Relational

13

14 of 20

Wrap up / Highlights

Cultural competence is the ability to understand, communicate with, and effectively interact across cultures.�
Evaluation can support diversity through equity and inclusion.�
Ethics in culturally responsive evaluations are procedural and relational.

14

15 of 20

Conclusion

With what have you learned and reflected in this module, how might you address cultural competency in your evaluation?

15

16 of 20

For further information

Garibay, C., & Teasdale, R. M. (2019). Equity and evaluation in informal STEM education. In A. C. Fu, A. Kannan, & R. J. Shavelson (Eds.), Evaluation in Informal Science, Technology, Engineering, and Mathematics Education. New Directions for Evaluation, 161, 87–106.

National Science Foundation. (2010). The 2010 User-Friendly Handbook for Project Evaluation. Retrieved from https://www.purdue.edu/research/docs/pdf/2010NSFuser-friendlyhandbookforprojectevaluation.pdf

American Evaluation Association. (2011). Public Statement on Cultural Competence in Evaluation. Fairhaven, MA: Author. Retrieved from https://www.eval.org/ccstatement

16

17 of 20

In the next module, you will be able to…

Describe the parts of a successful evaluation plan,
Understand and apply criteria to choose potential programs, and �
Identify an appropriate program to evaluate.

17

18 of 20

Stay connected

Visit us: https://research.ced.ncsu.edu/impactnc/

Dr. K.C. Busch

Assistant Professor of STEM Education

North Carolina State University | College of Education

kbusch@ncsu.edu

18

19 of 20

Credits

K.C. Busch

Lynn Chesnut

Regina Ayala Chávez

Aimee Fraulo

Kathryn T. Stevenson

Katy May

Lincoln Larson

Madeline Hinckle

19

20 of 20

20