1 of 35

QUALITATIVE RESEARCH METHODS

Technical Bootcamp

2 of 35

Objectives of this session

Understand how qual methods can be used across different phases of program/research cycle

Feel comfortable with some central (qual) research terms and approaches

Attain familiarity with the process of conducting and analyzing qual interviews

Appreciate that planning for analysis is imperative in planning for good measurement design, data collection (for many types of research)

8

3 of 35

Agenda

Presentation template – footer updates centrally

3

6 September 2021

1

Hallmarks of a qualitative approach

Qualitative approaches over the program/research cycle

From sampling to analysis: steps in conducting and analyzing interviews

2

3

4 of 35

Why qual?

“…people are motivated through the stories they are telling themselves at the time they make their decisions” and to understand people’s choices, you must examine their stories.” (Akerlof 2020)

5 of 35

Why qual? (examples)

5

IDinsight (with RAND) is conducting an RCT to understand whether setting small incentives for adolescents living with HIV in Uganda is more effective at improving medication adherence when incentives are conditional on reaching: a) a Fixed, clinically optimal threshold b) Assigned targets, based on reaching incremental subgoals leading up to the clinical threshold c) Participatory targets, where participants themselves can choose their targets and must ultimately reach the high clinical threshold.

We also collect qualitative data on potential pathways through which the intervention may have worked and ask a series of questions about how participants set their goals and their perceptions about goal attainability during each prize drawing round.

Quantitative finding (from prior small-scale pilot)

Participants who had very low adherence at baseline improved adherence the most when they were allowed to select subgoals on their own (Participatory group).

Qualitative findings

Revealed mechanisms related to motivational reinforcement

“The first time when I had chosen 80, I was glad to know that I had achieved

it, and I set another goal, higher than that because I wanted to see if I can test myself if I could meet it.”

6 of 35

Why qual? (examples)

6

Frye (2017) is a mixed methods study combining longitudinal survey data analysis with in-depth interviews with teachers, administrators, and youth to understand the link between sexual relationships and schooling exits in Malawi. The author adopted a recursive mixed-methods approach, moving repeatedly back and forth between longitudinal survey data and in-depth interviews and using insights gleaned from each type of data to inform my subsequent analysis of the other.

Quantitative finding

Survey data confirm the statistical association between sexual activity and girls school dropout
But it does NOT confirm narrative cultural expectations of the mechanisms (i.e. girls grow distracted and cannot learn, they miss school to be with their partners, and they fall pregnant).

Qualitative findings

Frye finds that teachers and parents act as though the narratives above above are inevitable and harshly punish or disinvest in girls who they believe are sexually active. It is these punitive actions that lead to dropout.
Policy implications are therefore very different in light of qual findings.

7 of 35

Hallmarks of a qualitative approach

8 of 35

Iteration

8

If you have mostly worked on large-n, quantitative projects, you’re familiar with a research process like this:

In qualitative work, rigor is pursued differently, because what counts as a “complete” dataset, or analysis completeness, differs when you are working with smaller samples and open-ended questions.

This means our linear, predictable process suddenly looks more like this (!):

9 of 35

Iteration

What iteration could look like:

Revising or dropping interview questions due to new information
Adjusting or supplementing the respondent sample to include previously unknown individuals with relevant knowledge or unique experiences
Going back to previous respondents to clarify responses
Expanding interview questions and/or research questions to new areas of interest
Adjusting hypotheses and theory, which may mean updating your codebook for further analysis (if your analysis method involves thematic coding)

Budgeting sufficient time for iteration is key

9

10 of 35

Positionality

Researcher’s positionality can affect:

The questions s/he asks and how frames them;
Her/his relations with those they research;
The interpretation s/he places on empirical evidence;
Access to data, organizations & outlets for research dissemination;
The likelihood that s/he will be listened to and heard about the findings

Mitigate risks of positionality by: independent coding, research transparency: description of who collected the data, how these individuals might have been perceived in the data-collect context, and what affect it may have had on the data

10

Positionality refers to the beliefs and value systems that are an inherent part of one’s perspective within the research process; the personal, gender, cultural and power perspective that researchers bring with them in into the field.

Researcher-as-instrument Along with the emphases on context, meaning, and the potential for researcher subjectivity, qualitative research is distinguished by the fact it places the researcher at the center of the data-gathering phase and, indeed, the researcher is the instrument by which information is collected. The closeness of the researcher to the research participants and subject matter instills an in-depth understanding which can prove beneficial to a thorough analysis and interpretation of the outcomes; however, this intimacy heightens concerns regarding the researcher’s ability to collect (and interpret) data in an objective, unbiased manner. Mitigating these effects is discussed here

Absence of “truth” With all the emphasis in qualitative research on reality and the human condition, it might be expected that qualitative inquiry is in the business of garnering “the truth” from participants. Instead of “truth,” the qualitative researcher collects information from which some level of knowledge can be gained. The researcher does not acquire this information and knowledge in a vacuum but rather in a context and, in this way, the research data are a product of various situational factors. For this reason, qualitative researchers do not talk about the “truth” of their findings but rather the “plausibility” of their interpretations. Plausibility is derived from achieving accuracy in the data collection process, accuracy in the absence of absolute “truth.”

“absence of truth.” This refers to the idea that the highly contextual and social constructionist nature of qualitative research renders data that is, not absolute “truth” but, useful knowledge that is the matter of the researcher’s own subjective interpretation. It is accuracy that qualitative researchers strive for when they craft their research designs to mitigate bias and inconsistency. For example, focus group moderators are trained to give equal attention to their group participants – allowing everyone an opportunity to communicate their thoughts – rather than bias the data – i.e., leading to inaccurate information – by favoring more attention on some participants than on others. A trained moderator is also skilled at listening for inconsistencies or contradictions throughout a discussion in order to follow up on each participant’s comments, asking for clarification, and ultimately coming away with an accurate “picture” of that participant in relationship to the topic as communicated in that particular space and time.

11 of 35

“Big-Tent” Criteria for qualitative research

11

Rich rigor	The study uses sufficient, abundant, appropriate, and complex • Theoretical constructs • Data and time in the field • Sample(s) • Context(s) • Data collection and analysis processes
Sincerity	The study is characterized by • Self-reflexivity about subjective values, biases, and inclinations of the researcher(s) • Transparency about the methods and challenges
Credibility	The research is marked by • Thick description, concrete detail, explication of tacit (nontextual) knowledge, and showing rather than telling • Triangulation • Peer debriefing • Member checking/ reflections
Ethical	The research considers • Procedural ethics (such as human subjects) • Situational and culturally specific ethics • Relational ethics • Exiting ethics (leaving the scene and sharing the research)

Of the 8 criteria listed in Tracy 2010, the following apply most to IDinsight-type qual research:

Credibility refers to the degree to which your research can be deemed responsible and accurate. It safeguards against subjective experiences, emotions, and perspectives that may otherwise obscure the true nature of your research subject.

riangulation

Triangulation is a powerful and effective strategy that significantly enhances the credibility of qualitative research. This technique involves using multiple sources or methods to collect and analyze data, thereby increasing the rigor and trustworthiness of the research findings. By combining different sources of information, such as interviews, observations, and focus groups, you can cross-check your results and ensure that your findings are consistent and reliable.

This approach also helps to identify potential biases or limitations in the research and provides a more comprehensive understanding of the phenomenon being studied. Employing triangulation allows you to ensure your research is based on diverse perspectives and experiences, strengthening the credibility and impact of your findings.

Member Checking

Another strategy used to build credibility is member checking. This is a powerful tool that validates the data’s accuracy and promotes a collaborative relationship between the researcher and the participants. This approach allows the participants to have a say in the research process, and their feedback is valuable in confirming the accuracy of the research findings.

Member checking is done in various ways, such as sharing transcripts, summaries, or the entire research report with the participants. The participants can then review the information and provide feedback to you on any discrepancies, inaccuracies, or missing data. This process helps establish credibility and builds trust and rapport between the researcher and the participants.

Additionally, member checking allows the participants to clarify or expand on their experiences, which can further enhance the richness and depth of the data.

Peer Debriefing

Peer debriefing is a valuable strategy for enhancing the credibility of qualitative research. It involves enlisting the help of other researchers, who are experts in the field, to review and provide feedback on the research process and findings. This process of critical evaluation helps to strengthen the research by identifying any weaknesses, biases, or limitations that may exist.

Peer debriefing is an effective tool for ensuring that the data collected is accurate, relevant, and reflects the participants’ experiences.

It’s important to note that peer debriefing is not just limited to reviewing research findings, but also includes evaluating the research process. This consists of the methodology, sampling strategy, and data collection techniques. By involving other researchers in this process, the research can be conducted with due diligence and rigor, ultimately increasing the overall quality of the study.

Thick Description

Thick description is a qualitative research strategy that provides a comprehensive and detailed description of the research setting and participants. This approach helps you better understand the context of your project, including the cultural, historical, and social factors that may influence the phenomenon under study. The help of detailed descriptions of the research context and participants also ensures your findings are grounded in the realities of the social world you are studying.

Thick descriptions also help you identify and explore your research topic’s complexities and nuances. For example, suppose you are studying the experiences of immigrants in a particular community; you may use thick descriptions to provide a detailed account of the cultural and social factors that shape their experiences. This helps uncover underlying patterns and themes that may not be immediately apparent and provides a deeper understanding of the respondents.

Reflexivity

Researchers’ biases and perspectives can influence the research process and findings in qualitative research, compromising the study’s credibility. Reflexivity is a strategy that aims to address this issue by encouraging you to reflect on your own biases and perspectives and account for them in the research process. This is achieved through keeping a research journal or memo, where you can reflect on your experiences, assumptions, and biases that may impact the research. Being transparent about your preferences and perspectives makes the research process more credible and trustworthy from the outset.

Furthermore, reflexivity can lead to new insights and perspectives that may not have been considered otherwise. By acknowledging and accounting for your biases and perspectives, the research process becomes more transparent, and the findings are more trustworthy. Reflexivity also helps you build stronger relationships with participants by showing a willingness to listen and learn from their perspectives.

12 of 35

Qualitative approaches over research / program cycle

13 of 35

Qual at IDinsight

13

IDinsight activity or service	Possible qual enhancements & stand-alones
Evidence & landscape review	Media or document content analysis High-level stakeholder interviews Review of relevant images Identification, study of key cases
Needs assessment / descriptive situation analysis / diagnostic	Observation / immersion Variety of interview strategies
Theory of change building, policy design support	Observation of how things currently (don’t) work Interviews (individual, group, etc) around barriers to, facilitators of change
Process, impact evaluation	Lots >> next slide

14 of 35

How to use qual in an IE or PE

14

Early in process	Refine research questions Build out a theory of change / generate hypotheses
Refining quant tools	Strengthen construct validity Cognitive interviewing on how people try to answer Qs Small-n qual to develop context-rooted close-ended questions for larger-scale quant
Alongside or as main show	Capture key variables in mult ways for triangulation Deeper exploration of experience of program/policy with longer interviews (many types), immersion/observations; can also use to solicit unintended consequences
Closing out well	Reflect on what did/not work with key subgroups Member-check interpretations of findings, co-interpret

Construct validity: "the degree to which a test measures what it claims, or purports, to be measuring.“
Cognitive interviewing: a technique used in piloting questionnaires during which respondents are asked to ‘think aloud’ and/or reflect on how they are answering questions, to help uncover misinterstandings and misinterpretations, over-reliance on heuristics, etc
Triangulation: A term borrowed from geometry & land surveying, with the premise that findings are made stronger through “validation of data through cross verification from two or more sources. In particular, it refers to the application and combination of several research methods in the study of the same phenomenon
Member-checking: (from Lincoln & Guba 1986): “the process of continuous, informal testing of information by soliciting reactions of respondents to the investigator’s reconstruction of what s/he has been told or otherwise found out and to the constructions offered by other respondents or sources; also, terminal, formal testing of the final case interpretations with a representative sample of key stakeholders.”

15 of 35

Additional uses of qual (Human-Centered Design)

15

16 of 35

16

17 of 35

From sampling to analysis: steps in conducting and analyzing interviews

18 of 35

Sampling to reach thematic saturation

18

Saturation: A core principle in qualitative research, which holds that a phenomenon has been adequately captured once no new concepts (data/codes) are being gathered with more interviews.

Illustration of reaching thematic saturation (at ~8 interviews) for the YouthWorks PH project:

Hennink and Kaiser 2021 found the following sample sizes reached thematic saturation:

9-12 interviews
4-8 focus group discussions (note: not participants per focus group)

19 of 35

Sampling purposively

19

Purposive sampling is the intentional selection of information-rich informants who are especially knowledgeable about or experienced with a phenomenon of interest. The objective of purposive sampling is not representativeness, but reaching saturation.

Types of purposive sampling designs (source: Palinkas et al. 2016):

Typical case: To illustrate or highlight what is typical, normal or average
Homogenous: To describe a particular subgroup in depth, to reduce variation, and simplify analysis
Snowball: To identify cases of interest from sampling people who know people that have similar characteristics
Criteria-based: Include all cases that meet some/all pre-specified criteria
Extreme or deviant-case: To illuminate the unusual (extreme successes or failures)
Stratified purposive - To capture major variations rather than to identify a common core, although the latter may emerge in the analysis

20 of 35

Sampling purposively (example)

20

21 of 35

Interview approach

Usually groups of 5-8 individuals with some shared characteristics
Used to uncover community norms and attitudes, and can unearth a variety of experiences
Not great for depth or very sensitive issues

Focus Groups

1-1 Interviews

Can be to explore a topic in-depth with an individual
Also used for someone who is an expert or specific stakeholder - ‘key informant’
Used to capture experience, stories, and individual opinions
Individuals may be less biased than in group setting

(Credit for the slide to Rachel LuSava from her slide deck “Unlocking Power in Qualitative Research: Why, When, How?”)

Presentation template – footer updates centrally

21

22 of 35

Creating interview guides

22

A semi-structured interview should read like a conversation to the interviewee. It should not read like a list of questions that map 1-1 to the research themes/topics you’d like to explore.

Some tips:

Avoid yes/no and close-ended questions
Keep wording neutral - insert neutral wording, unless intentionally provocative to spark a response. “Why do you think workplace gender equality is important? → “Do you think there are positive or negative effects of pursuing workplace gender equality”
Provide preambles - Especially important if the question is likely to be sensitive. This can reassure participants that they are not being singled out by being asked this topic, and that others share their views so they feel safe to express it.
Open with hearsay - Allows respondents to say things they might not say due to social desirability bias. “Some people think x. Others think y. We would like to know your thoughts…”
Provide space for talking about less-desirable behaviors. “When dealing with a sensitive behavior, provide space for it to be ‘natural’ to engage in less desirable behavior”

23 of 35

The life cycle of a qualitative question

23

24 of 35

Coding and analysis

24

“Code”—Differentiate, combine, categorize units of meaning within interview and across interviews

Use codes to generate descriptive and analytical insights

Present findings and analytical process in a transparent, logical manner.

25 of 35

Framework analysis

This is a common method at IDinsight of managing and analyzing qual data.

Characteristics:

Systematically managing and analyzing semi-structured interview data, allowing for associations within interviews as well as comparisons across interviews
Both assessing theories (our Theory of Change), and descriptively capturing phenomena
Using both deductive and inductive processes to code the data and to consolidate towards themes that can answer our research questions

25

(Gale 2013) “The Framework Method sits within a broad family of analysis methods often termed thematic analysis or qualitative content analysis. These approaches identify commonalities and differences in qualitative data, before focusing on relationships between different parts of the data, thereby seeking to draw descriptive and/or explanatory conclusions clustered around themes”

26 of 35

Inductive Coding vs Deductive Coding

Presentation template – footer updates centrally

26

6 September 2021

“Code”: A word or short phrase that symbolically assigns a summative, salient, essence-capturing, and/or evocative attribute for a portion of data.

“Coding”: the assignment of summative terms (a code) to sections of textual data.�

The application of codes developed from theories prior to data collection

The application of codes developed after data collection to capture patterns or associations that emerge from the data.

DEDUCTIVE CODING

INDUCTIVE CODING

27 of 35

Framework analysis steps

Steps:

Transcribe and Translate interviews
Data entry and organization into analysis matrix
Familiarization and disaggregation
First-cycle coding
Second-cycle coding
Categories, themes, and comparisons

27

Some of these steps may be faciliated/streamlined with AI tools for qual – (currently being refined by DSEM team as part of Lean Innovation grant)

28 of 35

Example coding tree

28

The code tree itself should be MECE. But your application of them may not. A sentence with many ideas can be tagged 4-5 times with many child codes, not counting the parent codes

29 of 35

Coding and analysis

29

Prepare for coding by 1) converting transcripts by adding context (required), and removing questions so transcripts read narratively (optional); & 2) revising your coding tree

30 of 35

Coding and analysis

30

Code the first time (first-round coding). Discuss with team. You or another person codes the second time (second-round coding) and/or have someone else review the code (review). Repeat.

31 of 35

Coding and analysis

31

Use the codes to group the data into categories, then themes.

A category is a grouping of codes which share similarities. For example, “family pressures” may be a category which include the codes husband-wife conflict, child-rearing, and parental expectations.

A theme is a higher level of categorization, which identifies the major elements of the data analysis. For example, the categories “family pressure” and “work requirements” may combine into the theme of “rigid work-life separation”.

Categories and codes will often be mapped onto the Research Questions or Theory of Change.

32 of 35

Coding method 1: Excel

32

33 of 35

Coding method 2: DEDOOSE

33

34 of 35

Which to choose?

34

Dedoose	Excel
Substantive	Substantive
•Can review ”excerpts” in full-transcript context •Cannot edit transcripts (textual data) after importing into project •Can handle (clunkily) non-textual data •Clunky cross-analysis between codes •Difficult to change coding tree once set up, but easy visualization •Enables simultaneous coding (service is cloud-based) •# of respondents & # of excerpts both easily attained •(Perhaps) more suitable for generating descriptive insights	•Cannot view the excerpt’s full transcript context inside software •Can edit transcripts (textual data) after importing into matrix •Can’t handle non-textual data •Easy cross-analysis between codes •Very easy to change coding tree, but no visualization • Difficult to code simultaneously (local storage unless web Excel or Gsheets) •# of excerpts (getting # of respondents requires more steps) •(Perhaps) more suitable to categorize data (for quant analysis)

35 of 35

Presenting findings

Display key themes drawn out for each research question
Provide information on how the theme was developed (what interview questions(s) led to the data, and categories or codes if the detail is helpful)
Indicate variation of responses within the theme, and, if relevant, illustrative quotes. These quotes may either represent common views, or—if deemed important—unique or extreme views. Take care in our selection and explanation to maintain respondent anonymity, and cite only the speaker’s respondent ID in footnote.
Do not attach numbers to our analysis (in general), as they often do not enrich qualitative results, but instead may lead to misunderstandings or misrepresentations of them, by implying these numbers may generalize to the population. Use qualitative modifiers “some” “a few” “most” “majority” instead.

35