Fundamentals of Survey Design: Questions
August 22, 2023
2
Icebreaker
In Breakout Rooms:
3
Session Norms
Adopt Growth Mindset
Produce Best Quality work
Uphold District Data Privacy
Maximize Engagement
Introduction (10 Minutes)
Writing survey questions (30 Minutes)
Background and Fact-Finding Recap (10 Minutes)
Designing the Survey (30 Minutes)
Closing (10 Minutes)
Agenda
4
4
Objectives
Data Fellows will identify and describe the key characteristics to design a survey.
Data Fellows will be able to identify the steps of Qualitative Research, Quantitative Survey, and Actionable Insights.
Success Criteria: By the end of the session, Fellows will be able to:
5
Objectives
Data Fellows will identify the steps related to how to use the survey design document.
Data Fellows will demonstrate the ability to apply objectives related to Information Needed, How to Ask Questions and How to Analyze Data collected from properly designed surveys.
Success Criteria: By the end of the session, Fellows will be able to:
6
7
Introduction to Writing Surveys Framework
2. Conduct background fact-finding
3. Design survey plan
4. Make questions clear
Adapted from The Complete Guide to Writing Questionnaires: How to Get the Information for Better Decisions
5. Make questions answerable
6. Make questions easy
7. Make questions unbiased
8. Pretest survey
Today’s focus
Background and Fact Finding Recap
In the chat:
Big Three Questions:
Example Survey: Teacher Confidence in HQIM
Respondents: Teachers in Lost Pines ISD
Sponsors’ Goal: Determine the extent to which teachers are confident in implementing Amplify and Eureka curricula.
Fundamentals of Survey Design: Questions
Vignette Scenario
Directions
Vignette
Michael’s nervous knock got Angela’s attention.
Angela nodded for him to come into her office.
Michael sighed and slumped into the seat.
“What’s wrong?”
“I just got out of a strategy session with the survey committee.” Michael exhaled.
Angela arched her brow and nodded.
“I’m concerned that the committee’s questions won’t get us the data we need to make decisions.”
“Can you show me the questions?”
Michael messaged the questions to Angela.
Angela read aloud a couple of the questions.
‘All staff and administrators share a vision for professional development.’
‘Teachers should do whatever is necessary to attend professional development.’
Angela smiled.
“Your Cheshire cat grin tells me everything.”
Angela nodded. “Let’s jam out on designing the survey design plan.”
“When?”
“I can do it now. The survey plan will empower you to go back to the steering committee with a document for more explicit connections.”
Michael smiled.
Vignette
Michael’s nervous knock got Angela’s attention.
Angela nodded for him to come into her office.
Michael sighed and slumped into the seat.
“What’s wrong?”
“I just got out of a strategy session with the survey committee.” Michael exhaled.
Angela arced her brow and nodded.
“I’m concerned that the committee’s questions won’t get us the data we need to make decisions.”
“Can you show me the questions?”
Michael messaged the questions to Angela.
Angela read aloud a couple of the questions.
‘All staff and administrators share a vision for professional development.’
‘Teachers should do whatever is necessary to attend professional development.’
Angela smiled.
“Your Cheshire cat grin tells me everything.”
Angela nodded. “Let’s jam out on designing the survey design plan.”
“When?”
“I can do it now. The survey plan will empower you to go back to the steering committee with a document for more explicit connections.”
Michael smiled.
What problem is the team trying to solve? Why is it important to solve it?
Survey Design Overview
What is a Survey?
"A survey is a 'conversation' between you and hundreds, and sometimes thousands of diverse respondents." – David F. Harris
The Complete Guide to Writing Questionnaires:
How to Get Better Information for Better Decision
16
Steps to survey design
17
Good Surveys
Unlike a live conversation, you are not able to clarify ambiguities.
Designing the Survey
Framework for Designing Survey Questions
How to Ask Questions
How to Analyze Data
Make questions clear, answerable, easy
Ensure questions have only one answer
Stem and answer choices must match
Determine method for data collection
Determine visuals based on the data you will collect
Conduct Cognitive Interview
Brainstorming session
Allows you to think about the information you will need from the questions you are answering
In our workbook, Column A
Gather Information Needed
20
Designing the Survey
Tool: Survey Workbook
Framework for Designing Survey Questions
How to Ask Questions
How to Analyze Data
Make questions clear, answerable, easy
Ensure questions have only one answer
Stem and answer choices must match
Determine method for data collection
Determine visuals based on the data you will collect
Brainstorming session
Allows you to think about the information you will need from the questions you are answering
In our workbook, Column A
Gather Information Needed
Let’s Practice
3 min solo:
Let’s Practice
5 min solo:
Let’s Practice
8 min Breakout Rooms
Debrief
Framework for Designing Survey Questions
How to Ask Questions
How to Analyze Data
Make questions clear, answerable, easy
Ensure questions have only one answer
Stem and answer choices must match
Determine method for data collection
Determine visuals based on the data you will collect
Brainstorming session
Allows you to think about the information you will need from the questions you are answering
In our workbook, Column A
Gather Information Needed
2 min solo
How to ask Questions
28
Make Questions Clear
Limit qualifiers to one
The PD I received this year was timely and helpful.
“PD was helpful, but too late in the year.”
“Which session are they talking about? There were 6 and they were all different.”
The PD we received in August on Eureka Math was helpful.
Initial Question
Participants’ Thoughts
Improved Question
29
Make Questions Answerable
Use specificity
I regularly receive feedback on my lesson internalization.
“My principal provided me feedback at the beginning of fall every week, but now it’s every month…is that regularly?”
In the past 30 days, how many times have you received feedback on your lesson internalization?
Initial Question
Participants’ Thoughts
Improved Question
30
Make Questions Easy
Use clear language; use simple sentences
Thinking of your typical student, what percentage of them have improved their knowledge because of the HQIM you have implemented in class?
“Uhh..I have not thought about this. I’ll make up an answer. I have things to do.”
“What’s typical even mean?”
“What is HQIM? I teach an elective and create my own curriculum…”
Only provided to those who use HQIM:
Think of your students with the ten lowest grades. Out of those students, please share your best guess: how many have you seen respond well to HQIM?
Initial Question
Participants’ Thoughts
Improved Question
31
Make Sure Questions have only one Answer
Avoid using “and;” instead, split the question into two parts
I have a strong understanding of these instructional materials and feel comfortable using them.
Yes…I have a strong understanding and feel comfortable…
Hmm..I have a strong understanding, but I don’t really feel comfortable because I disagree with some elements.
I feel comfortable using the instructional materials.
Initial Question
Participants’ Thoughts
Improved Question
32
Question Stem and Answer Choices Must Match
Ensure you select the appropriate scale of measurement
My instructional coach and I meet frequently.
(Strongly Agree → Strongly Disagree)
“I think we meet frequently…but what if my coach doesn’t agree with me?”
“We meet frequently but sometimes we meet for other reasons…”
Feedback sessions with my instructional coach have improved my teaching effectiveness.
(Strongly Agree → Strongly Disagree)
Initial Question
Participants’ Thoughts
Improved Question
33
Avoid Question Bias
Avoid influencing respondents with your own assumptions
Because students are far behind, teachers spend nearly all of their time on standards from earlier grades.
“Not all students are behind.”
“I spend about 60% of my time on standards from earlier years…is this nearly all of my time?”
I spend more than 30% of my time teaching previous grade-level standards.
True/False
Initial Question
Participants’ Thoughts
Improved Question
Let’s Practice
5 min solo:
35
Test your Questions: Cognitive Interview
Sample of 5-10 people
Don’t focus on their response; focus on how they arrived at their response
Questions to Ask
Understand if your question is clear and answerable
Understand if your question has only one answer and avoids bias
Understand if your questions are easy and have only one answer
36
Model: Cognitive Interview
Questions to Ask
Let’s Practice
~ 10 min Breakout Rooms
Framework for Designing Survey Questions
How to Ask Questions
How to Analyze Data
Make questions clear, answerable, easy
Ensure questions have only one answer
Stem and answer choices must match
Determine method for data collection
Determine visuals based on the data you will collect
Brainstorming session
Allows you to think about the information you will need from the questions you are answering
In our workbook, Column A
Gather Information Needed
39
How to Analyze the Data
Estimate populations:
Differences between groups:
Associations between variables:
Closing
41
Upcoming Events
Image
Office Hours: Thursday, August 24
Community of Practice: Thursday, August 31