Fundamentals of Survey Design: Qualitative Data
August 8, 2023
2
Meet our New Team Member!
Beri Tamsen
3
Icebreaker
In Breakout Rooms:
What is your favorite memory from this summer?
4
Session Norms
Adopt Growth Mindset
Produce Best Quality work
Uphold District Data Privacy
Maximize Engagement
Introduction (10 Min.)
Top-Line Report (20Min.)
Fundamentals of Survey Design (Qualitative vs. Quantitative Data) and Potential Roadblocks (20 Min.)
Designing Questions for Research Sponsors and Potential Respondents (30 Min.)
Closing (10 Min.)
Agenda
5
5
6
Introduction to Writing Surveys Framework
2. Conduct background fact-finding
3. Design survey plan
4. Make questions clear
Adapted from The Complete Guide to Writing Questionnaires: How to Get the Information for Better Decisions
5. Make questions answerable
6. Make questions easy
7. Make questions unbiased
8. Pretest survey
Today’s focus
7
Where we are, Where we’re going
August
September
Objectives
Data Fellows will identify and describe the key characteristics of qualitative survey data, such as open-ended questions, narrative responses, and subjective opinions.
Success Criteria: By the end of the session, Fellows will be able to:
8
Objectives
Data Fellows will create a plan to effectively engage with stakeholders to gather relevant background information, such as the purpose of the survey, target audience, desired outcomes, and specific research objectives, ensuring a clear understanding of the context and goals for survey design.
Data Fellows will demonstrate the ability to translate stakeholder requirements and objectives into survey design elements, such as selecting appropriate question types, designing logical survey flow, and incorporating relevant demographic or contextual variables, resulting in a survey instrument that aligns with the intended analysis and facilitates meaningful data collection.
Success Criteria: By the end of the session, Fellows will be able to:
9
Fundamentals of Survey Design and Potential Roadblocks
Fundamentals of Survey Design Qualitative Data
Vignette Scenario
Directions
Angelique clicked to slide seventy-eight. “Now let’s see how respondents answered question forty-five on our teacher professional development survey.”
“Hold on a minute please.” Marco rubbed his eyes to re-energize.
“Do you want me to go over any previous points?”
“No. We all appreciate your attention to detail. But what can we implement based on the responses?”
“This is the last slide and we can dig into the recommendations. It is important to share with you the production of the survey, the output of the survey and then end with recommendations.”
The seven-person committee waited for Marco, the committee lead to respond to Angelique, the survey researcher. Some committee members clasped their hands. Other members laid down the pencils on their pads.
“This presentation is not going where we envisioned.”
Angelique’s cheeks flushed. “I provided a copy of the questions before the survey launched. Could you tell me what I am missing?”
“You did an excellent job crunching the numbers but I am bewildered. How many more hours of professional development should we offer? Which topics would be most relevant to teachers? When is the best time to offer professional development?”
“My understanding from conversations with the committee is that you need to know as much as possible about professional development from the teacher’s perspective.”
“I think we used the wrong blueprint to build the survey.”
“What would you like to do at this point?”
We need three volunteers to play the characters.
Reflect: what is the disconnect between Angelique and Marco?
Angelique clicked to slide seventy-eight. “Now let’s see how respondents answered question forty-five on our teacher professional development survey.”
“Hold on a minute please.” Marco rubbed his eyes to re-energize.
“Do you want me to go over any previous points?”
“No. We all appreciate your attention to detail. But what can we implement based on the responses?”
“This is the last slide and we can dig into the recommendations. It is important to share with you the production of the survey, the output of the survey and then end with recommendations.”
The seven-person committee waited for Marco, the committee lead to respond to Angelique, the survey researcher. Some committee members clasped their hands. Other members laid down the pencils on their pads.
“This presentation is not going where we envisioned.”
Angelique’s cheeks flushed. “I provided a copy of the questions before the survey launched. Could you tell me what I am missing?”
“You did an excellent job crunching the numbers but I am bewildered. How many more hours of professional development should we offer? Which topics would be most relevant to teachers? When is the best time to offer professional development?”
“My understanding from conversations with the committee is that you need to know as much as possible about professional development from the teacher’s perspective.”
“I think we used the wrong blueprint to build the survey.”
“What would you like to do at this point?”
What was the disconnect between Angelique and Marco?
Poll: Survey Experience
15
Research Roadblocks
Please expound upon a roadblock that resonates with you!
Goal for August Sessions
Survey design best practices to minimize the following:
Designing Questions for Research Sponsors and Potential Respondents
18
Reflection
Survey design must start with qualitative research. Why?
19
Gathering Qualitative Data
There are 2 key steps in gathering qualitative data
2. Ask potential respondents sample questions to test question validity.
Step 1: Plan your research
Who do you want to ask? What do you want to ask?
Research Sponsors
Research Sponsors: Commission and approve research launch, analysis, and publication
Research Sponsors can be district administrators, RSSP lead, etc.
Questions for Research Sponsors
Begin with the end in mind
Three questions to frame the survey before undertaking the task of writing questions.
Step 1: Plan your research
Who do you want to ask? What do you want to ask?
Research Sponsors
Research Sponsors: Commission and approve research launch, analysis, and publication
Research Sponsors can be district administrators, RSSP lead, etc.
Questions for Research Sponsors
Begin with the end in mind
Three questions to frame the survey before undertaking the task of writing questions.
If your research sponsors cannot answer all three questions, you likely do not need a survey.
Step 2: Test your questions
Qualitative Research overview
Let’s Practice
Example Survey: ISD Parent Satisfaction Survey
Respondents: Parents of students in school district
Sponsors’ Goal: Determine the extent to which parents are satisfied having their children enrolled in Lost Pines ISD.
Look at each question on the table and reflect on the questions located on your Note-Catcher.
Example Survey: ISD Parent Satisfaction Survey
Respondents: Parents of students in school district
Sponsors’ Goal: Determine the extent to which parents are satisfied having their children enrolled in Lost Pines ISD.
Qualitative Topic Guide | Translation | Survey Sample Question |
Tell me why you chose to enroll your child in our school district. | Goal: Get a list of reasons why parents enroll children into Lost Pines ISD | Below are reasons why parents enroll their children in Lost Pines ISD. Please check all that apply to you. Reason 1 Reason 2 Reason 3 Other (please specify) |
Example Survey: ISD Parent Satisfaction Survey
Respondents: Parents of students in school district
Sponsors’ Goal: Determine the extent to which parents are satisfied having their children enrolled in Lost Pines ISD.
Qualitative Topic Guide | Translation | Survey Sample Question |
What is your favorite part about having your children enrolled in Lost Pines ISD? (Probe: what makes this your favorite part?) What is your least favorite part about having your children enrolled at Lost Pines? (Probe: what makes this your least favorite part?) | Goal: create a relevant rating scale | Please rate each statement on a scale from Least Favorite to Most Favorite: Statement 1 Statement 2 Statement 3 Other (please specify): |
Example Survey: ISD Parent Satisfaction Survey
Respondents: Parents of students in school district
Sponsors’ Goal: Determine the extent to which parents are satisfied having their children enrolled in Lost Pines ISD.
Qualitative Topic Guide | Translation | Survey Sample Question |
What support from the district would increase your satisfaction with teachers at your campus? (Probe: what would you need to make this possible?) | Goal: create a list of ways to increase parent satisfaction with teachers | Please select the statements below that would increase your satisfaction with the teachers at your campus. Please check all that apply: Statement 1 Statement 2 Statement 3 Other (please specify): |
Let’s Practice
In Breakout Rooms:
Example Survey: Teacher Confidence in HQIM
Respondents: Teachers in Lost Pines ISD
Sponsors’ Goal: Determine the extent to which teachers are confident in implementing Amplify and Eureka curricula.
Whole Group Recap
Qualitative Top-Line Report
30
Create the Top-Line Report
Let’s Practice
In Breakout Rooms:
Closing
Did we Accomplish Our Objectives?
Data Fellows will identify and describe the key characteristics of qualitative survey data, such as open-ended questions, narrative responses, and subjective opinions.
Data Fellows will demonstrate the ability to differentiate quantitative survey data by recognizing its main features, such as closed-ended questions, numerical responses, and the ability to be measured and analyzed statistically.
Success Criteria: By the end of the session, Fellows will be able to:
33
Did we Accomplish Our Objectives?
Data Fellows will create a plan to effectively engage with stakeholders to gather relevant background information, such as the purpose of the survey, target audience, desired outcomes, and specific research objectives, ensuring a clear understanding of the context and goals for survey design.
Data Fellows will demonstrate the ability to translate stakeholder requirements and objectives into survey design elements, such as selecting appropriate question types, designing logical survey flow, and incorporating relevant demographic or contextual variables, resulting in a survey instrument that aligns with the intended analysis and facilitates meaningful data collection.
Success Criteria: By the end of the session, Fellows will be able to:
34
35
Upcoming Events
Image
Office Hours: Thursday, August 10
Thursday, August 24
Synchronous Session: Tuesday, August 22
Community of Practice: Thursday, August 31
EOM Report will be in your inbox by Friday!
Exit Ticket