Psychometric Evaluation of Measures�in Studying Diverse Populations

Presenter: Nari Yoo

PhD Student, NYU Silver School of Social Work

​

Mar 29, 2023

Conducting Research with Diverse Populations

Before we begin: What does “Statistics” mean to you?

2

Usefulness of the Quantitative Research

• Description – Describe the phenomenon to the policy makers and the public
• Explanation – Find the mechanism (why and how) on predictors, mediators, moderators, …
• Prediction – Predict the potential problems and prevent them

What kind of research questions do you have?

Choosing variables

Building Hypotheses

Measurement

3

Basic Terms

• Culture: the set of beliefs, practices, learned behavior and moral values that are passed on, from one generation to another.
• Society: an interdependent group of people who live together in a particular region and are associated with one another.

​

• There may be various societies within a country, and there may also be cultures shared by various communities within it.

​

https://keydifferences.com/difference-between-culture-and-society.html

4

Cultural Universals and Particulars

5

Potential Biases in Cross-Cultural Research

• It’s developed and tested in specific period and on specific populations �(at that time might be universal, but it’s not now!)

​

The distortion of test scores due to… (van de Vijver and Tanzer 2004)

• Construct bias: differences in the theoretical constructs being measured across cultures
• if a test measures intelligence, but the definition of intelligence varies across cultures, then the test may not be measuring the same construct in each culture.
• Method bias: differences in the methods used to administer tests across cultures
• if a test is administered differently in one culture than another (e.g., timed vs. untimed), this could affect the scores obtained from each culture.
• Item bias: distortions at the item level, where certain items on a test may have different psychological meanings across cultures
• if an item on a self-report inventory is partly determined by social desirability in one cultural group, then including that item could invalidate comparisons of total test scores across cultures.

​

6

Potential Biases in Cross-Cultural Research

• Example: Review of Culture-Specific elements within stigma measures (Yang, Thornicroft, Alvarado, Vega, Link, 2014- Int. J of Epidemiology)`
• Literature Review (1990- 2012) of MI stigma in non-Western settings
• 5,292 abstracts-- 196 articles
• 77% of studies used translations of Western (U.S., U.K.) developed measures
• ONLY 2% of studies used quantitative stigma measures developed within a non-Western European cultural group
• 16.8% of studies used qualitative methods; demonstrated culture-specific stigma processes

7

Things to contribute cultural insensitivity

1. The development of content validity based on experts' rational analysis of concepts while neglecting cultural understandings.
2. Linguistic translations that try to conform to the exact terms of standardized instruments.
3. The uncritical transferring of concepts across cultures.

​

(Rogler 1999)

1. Examine procedural norms and keeping them from suppressing, biasing, or deflecting cultural understandings.
2. Engage in a dialogue on the intricate connections between culture and customary methodologies.

8

Discussion

• Are there any scales you care about that are likely to have cultural differences?
• How do those measures could differ from culture to culture?

9

Back to Statistics: �Introduction to Psychometric Measurement

• In public health and epidemiology, researchers are more likely to use econometrics or biostatistics (e.g., survival analysis) since the “causality” factor is so crucial to their findings. If you are interested in the effectiveness of interventions, causal inference and experimental research design to prove causation before/after intervention are essential.
• Ref: Jeffries, N., Zaslavsky, A. M., Diez Roux, A. V., Creswell, J. W., Palmer, R. C., Gregorich, S. E., … & Breen, N. (2019). Methodological approaches to understanding causes of health disparities. American journal of public health, 109(S1), S28-S33.
• Columbia Epidemeology – Public Health Methods�

10

Back to Statistics: �Introduction to Psychometric Measurement

• Social work or education scholars are more likely to use psychometrics since they are interested in subjective concepts considering populations of interest.
• However, there are exceptions: social welfare policy scholars, educational policy scholars, and interventionists tend to use more econometrics than psychometrics in their research. In recent years, both lines of statistical methods have been developed by integrating computational approaches (e.g., introducing SEM for machine learning).

​

​

11

Back to Statistics: �Introduction to Psychometric Measurement

• Psychometrics: if you are interested in subjective variables (e.g., mental health, stigma, well-being…)
• Utilize psychometric measurement by surveying or collecting primary dataset 
• Structural equation modeling is a combination of psychometric measurement (measurement model part) and a series of regression (structural model part)

• Econometrics: if  you are interested in objective variables (e.g., policy outcome, demographic outcome…)
• Mostly utilize secondary data, observational data
• Causal inference (instrumental variable, difference in difference, propensity score matching …)

12

Back to Statistics: �Introduction to Psychometric Measurement

https://conjointly.com/kb/true-score-theory/

Classical Measurement Theory

13

Back to Statistics: �Introduction to Psychometric Measurement

14

Types of Equivalence

15

Case Study: 15-item FFMQ-15

• The 15-item FFMQ (FFMQ-15) was developed by Baer et al. (2008) and includes three items for each facet.
• The factor structure and psychometric properties of the FFMQ-15 were tested by Gu et al. (2016).
• They found that the factor structure of the FFMQ-15 was consistent with that of the FFMQ-39.
• However, most (97.5%) of the sample was white.

​

Heeren, A., Lannoy, S., Coussement, C., Hoebeke, Y., Verschuren, A., Blanchard, M. A., ... & Gierski, F. (2021). A network approach to the five-facet model of mindfulness. Scientific Reports, 11(1), 1-11.

Gu, J., Strauss, C., Crane, C., Barnhofer, T., Karl, A., Cavanagh, K., & Kuyken, W. (2016). Examining the factor structure of the 39-item and 15-item versions of the Five Facet Mindfulness Questionnaire before and after mindfulness-based cognitive therapy for people with recurrent depression. Psychological assessment, 28(7), 791.

16

Case Study: Sample

17

Scale: 15-item Five-Facet Mindfulness Questionnaire

• Observing items: 1, 6, 11.
• 1. When I take a shower or a bath, I stay alert to the sensations of water on my body.
• 6. I notice how foods and drinks affect my thoughts, bodily sensations, and emotions.
• 11. I pay attention to sensations, such as the wind in my hair or sun on my face.

​

• Describe items: 2, 7R, 12.
• 2. I’m good at finding words to describe my feelings.
• 7. I have trouble thinking of the right words to express how I feel about things.*
• 12. Even when I’m feeling terribly upset I can find a way to put it into words.

18

Scale: 15-item Five-Facet Mindfulness Questionnaire

• Acting with awareness items: 3R, 8R, 13R.
• 3. I don’t pay attention to what I’m doing because I’m daydreaming, worrying, or otherwise distracted.*
• 8. I do jobs or tasks automatically without being aware of what I’m doing.*
• 13. I find myself doing things without paying attention.
• Non-judging items: 4R, 9R, 14R.
• 4. I believe some of my thoughts are abnormal or bad and I shouldn’t think that way.*
• 9. I think some of my emotions are bad or inappropriate and I shouldn’t feel them.*
• 14. I tell myself I shouldn’t be feeling the way I’m feeling.*
• Non-reactivity items: 5, 10, 15.
• 5. When I have distressing thoughts or images, I “step back” and am aware of the thought or image without getting taken over by it.
• 10. When I have distressing thoughts or images I am able just to notice them without reacting.
• 15. When I have distressing thoughts or images I just notice them and let them go.

19

Problem Statement

• Although mindfulness has philosophical origins in Buddhist culture from the East, its conceptualization has been started in the Western culture and is being exported back to Eastern cultures.
• Scales of mindfulness can be influenced by an individual's cultural background. For example, individualism-collectivism, well known as the difference between East and West, and the degree of self-expression can affect mindfulness.
• Several studies have proven that they have different factor structures as a result of cross-cultural validation in China and Thailand.
• However, in the U.S., the existing mindfulness scale validation -- especially FFMQ-15 -- was mainly conducted on the Female Caucasian Sample.

​

• Validation in an Asian-American sample of the 15-item FFMQ is needed.

20

Factor Analysis: Eigenvalue

21

Factor Analysis: Parallel Analysis

22

Factor Analysis: Factor Loading

oblique promax

(blanks represent abs(loading)<.29)

23

Factor 1: Negative Self-reflection

• Acting with awareness items: (original scale) 3R, 8R, 13R.
• 3. I don’t pay attention to what I’m doing because I’m daydreaming, worrying, or otherwise distracted.*
• 8. I do jobs or tasks automatically without being aware of what I’m doing.*
• 13. I find myself doing things without paying attention.
• Non-judging items (original scale): 4R, 9R, 14R.
• 4. I believe some of my thoughts are abnormal or bad and I shouldn’t think that way.*
• 9. I think some of my emotions are bad or inappropriate and I shouldn’t feel them.*
• 14. I tell myself I shouldn’t be feeling the way I’m feeling.*
• Describe item (original scale): 7R.
• 7. I have trouble thinking of the right words to express how I feel about things.*

24

Factor 2: Positive self-reflection

• Observing items (original scale): 1, 6, 11.
• 1. When I take a shower or a bath, I stay alert to the sensations of water on my body.
• 6. I notice how foods and drinks affect my thoughts, bodily sensations, and emotions.
• 11. I pay attention to sensations, such as the wind in my hair or sun on my face.
• Describe items (original scale): 2, 12.
• 2. I’m good at finding words to describe my feelings.
• 7. I have trouble thinking of the right words to express how I feel about things.*
• 12. Even when I’m feeling terribly upset I can find a way to put it into words.
• Non-reactivity items (original scale): 5, 10, 15.
• 5. When I have distressing thoughts or images, I “step back” and am aware of the thought or image without getting taken over by it.
• 10. When I have distressing thoughts or images I am able just to notice them without reacting.
• 15. When I have distressing thoughts or images I just notice them and let them go.

25

Reliability Analysis

26

International Validation Procedure

• International validation procedures: a set of guidelines and standards that ensure the cultural fit and measurement invariance of psychological instruments across different populations and cultures
• a rigorous process of translation, adaptation, and validation of the instrument to ensure that it is culturally appropriate and valid for use in the intended population

�

27

International Validation Procedure

• The International Test Commission (ITC) guidelines for translating and adapting tests
• 1. Forward translation: The M-CTS was translated from English to Spanish by two independent translators who were native Spanish speakers.
• 2. Back-translation: The translated version was back-translated into English by two independent translators who were native English speakers.
• 3. Expert review: A panel of experts reviewed the translations and back-translations to ensure that the meaning of the items was preserved and culturally appropriate.
• 4. Pilot testing: The adapted version was pilot-tested with a sample of 30 Mexican adolescents to identify any issues with comprehension or cultural relevance.
• 5. Confirmatory Factor Analysis (CFA): A CFA was conducted to evaluate the factor structure of the adapted version and compare it with the original version.
• 6. Evaluation of Construct and Known Groups Validities: The construct validity of the adapted version was evaluated by examining its correlation with other measures of dating violence, while known-groups validity was assessed by comparing scores between groups with different levels of dating violence.

28

Conclusion: Measurement Issues

• Cultural Adaptation of existing measures?
• Translation of existing measures?
• Examination of measurement invariance?
• Development of new measures?

29

Reference

30

Questions?

Thanks!

Contact: nari.yoo@nyu.edu

Personal blog: nariyoo.com