1 of 13

Designing Effective Surveys

Evaluation, Assessment & Data Center

https://tinyurl.com/sl9jeqy

2 of 13

Learning Objectives

Start with the “Why?”
Identify target audience(s) and ways to disseminate your findings
Identify the “Big” questions the survey results will answer (including appropriate data analysis methods)
Learn how to write good survey items - appropriate item types and the appropriate number of items to use for each of the “Big” questions
Apply strategies for effective survey form creation and survey administration (look and feel, piloting, timing, etc.)

3 of 13

Lauren:

When it comes to survey planning, this is the roadmap we use

We start with the Why and work our way around the diagram until we have a good idea of what’s needed and how we can deliver

The blue section is the general aims of the survey and its intended purpose (The Planning Phase)
The orange section are the technical components/nuts and bolts of the survey (The Collecting Phase)
The green section illustrates what to do with the survey data (The Reporting/Decision Making Phase)
Each step of the green section is directly aligned with each step in the blue section. And when we get to the green steps we want to make sure we’re addressing all the same points we set for ourselves in the blue section.
Each step is dependent on the step before it, so if we want that last step in green to go well or to be well received, everything that comes before it has to also go well.
So, when we’re asked to put together a survey, you can see why we ask so many questions instead of just getting to work.

4 of 13

What is the survey’s purpose? What do you hope to learn?

Improve a program
Demonstrate results of a program
Consider: Is there another way to find out?

What do you want to know (general ideas)?

Participant reactions/satisfaction
Teaching and facilitation
Reasonable outcomes
Future programming
Participant background

Who gets the findings/Who has the answers?

Program leadership and funders
Consider: Survey takers, those impacted by the program or the loss of the program, other key stakeholders?

Lauren:

Why?

Survey’s purpose and hope to learn
Decided by the program leadership or whoever decided “we should do a survey”
Consider: Is there another way to find out instead of a survey?

Direct observation, small or large group discussion, personal interviews, a previous survey, document review (meeting agendas/minutes, sign-in sheets, photo or video evidence), personal anecdotes, student data

Who gets the findings? Who should know the results of the survey?

Obviously, program leadership and funders
Consider: Who else should be informed? Survey takers/participants, program staff, anyone else?
Decided by the program leadership

Big Questions (large constructs)

(large constructs) - Not the survey items but the types of information you can usually collect in a survey
Participant reactions: do they feel positively or negatively about something, do they like or not like something, are they satisfied - reactions are not the same as outcomes

Reactions to the session - content and delivery
Reactions to how it was carried out - administration, facilities, and logistics

Teaching and facilitation: how well you are teaching and facilitating their learning

Knowledgeable, prepared, able to engage participants, good use of time, how well did they answer questions

Outcomes are the things you intended to change by doing the program or event

Commonly: perceived changes in knowledge, attitude/opinion, skill development, and intended behavior
Also ask about unintended or unexpected outcomes
Any other benefits from participating?
These have to be reasonable given how the program is implemented, the length of their participation, what the program specifically covered, how much prior experience the participants had in that area before coming in, etc.
Anothing thing to consider: surveys are self-report, after the fact, questionnaires that ask people for their perceptions on outcomes and should not be used to measure actual learning or change

Future programming: what are participants’ need related to the current program and areas of interest for future sessions
Participant background is who is taking your survey

Are they from a school district, a school, organization, agency, etc
What is their role? Superintendent, administrator, teacher, parent, community member,

Decided by program leadership
Ok to pick 1 of these, all of these, or any combination in between

The right answers will depend on your program/event, your reason for the survey, your audience, and your information needs

Usually, all of these (why, who, and big questions) can be answered at the initial meeting and they don’t have to be answered by a group or committee, sometimes it’s just 1 or 2 people. But it’s helpful to have these on hand to inform other people of the survey you’re planning to do and if you’re bringing anyone else on board to work on the survey, to get them up to speed

When do you do this?

As part of planning the program/event

5 of 13

Align to Big ?s
Audience characteristics
Bias
Minutes to complete
Need to know vs. Nice to know
Be specific & clear
Easy/hard/easy
Intro paragraph

Use Resources:

Lauren

Write survey items:

Step1: develop the questions:

Align to BIG ?s
What are the indicators of that construct?

FOR EXAMPLE: if Big Question—>are teachers implementing program? Then items maybe: 1) how comfortable/confident are you using the curriculum? 2) How many lessons have you taught from the curriculum? 3) How often do you use the curriculum? (every day, weekly, monthly, etc.)

Guard against bias (don’t ask leading questions)
Double-barreled questions, etc.

Step 2: Adapt to audience

Use of terms used by the survey takers
Other audience characteristics: language, education level, cultural differences, interest level, etc.

Step 3: Reduce burden on survey taker - don’t make them feel bad, stressed or have to interpret what you are saying

Minutes to complete - you want to keep it short - prevent survey fatigue
Nice to know vs. Need to know - Be choosy, obtain the most amount of data in the least amount of time, limit data collection to information you will use, protect privacy and confidentiality - don’t collect identifying information unless you absolutely need it
Be specific & clear - match your audience and what they know and understand - definitions are helpful, eliminate bias, cultural bias, anti-negative bias
Item order - start off with easy questions (demographics) then progress to harder items, keep open-ended/free response questions to a minimum and save for the end
Intro paragraph - a good intro paragraph will help participants decide if they want to take your survey or not. Try to predict what concerns a survey taker would have and address them immediately. Keep it short, casual, and inviting

Include instructions to survey proctors

Design survey form:

Eye strain (font size, bolding of important words, layout, color use, etc.)
Logo and color

Visually appealing
Make good use of space on the page
The survey should not look like it will take too long or require too much thought

Translation

how will participants access those translated surveys?
How will written comments be translated back into English and analyzed

Paper vs. Online

Do you KNOW participants have the ability and capacity to complete an online survey - WiFi/internet connection, device, tech savviness
Survey logic/survey paths

Pilot survey:

Proofread
Critical friend/representative survey taker - Do they understand the survey questions as you intended? How much time does it take to complete survey? How do testers “feel” after having taken the survey?
Go back and forth between survey design and piloting until you’re comfortable with the end result
Get necessary approvals
Give yourself plenty of time to complete these steps - how long depends

Administer survey:

When to distribute to maximize response rate?

at the close of the event with a captive audience - paper exit ticket
Online

collect participants’ email address
How will you track responses and who to send reminders to?

Work around/avoid major events (holidays, assemblies, etc) and conflicting with other data collection efforts (state testing or a competing survey)

Allow for time to complete

Paper - dedicated time in the schedule
Online - as close to the program’s/event’s conclusion as possible while it’s still fresh in their memory

Give survey takers an appropriate amount of time

Incentives - if there is room in the budget and you give yourself the time and staffing you need to deliver on the incentive

REMEMBER - Plan ahead and plan early

Bad survey = bad data
you can only send out the survey once

6 of 13

Eye strain

Logo and color

Translation

Paper vs. Online

Lauren

Write survey items:

Step1: develop the questions:

Align to BIG ?s
What are the indicators of that construct?

FOR EXAMPLE: if Big Question—>are teachers implementing program? Then items maybe: 1) how comfortable/confident are you using the curriculum? 2) How many lessons have you taught from the curriculum? 3) How often do you use the curriculum? (every day, weekly, monthly, etc.)

Guard against bias (don’t ask leading questions)
Double-barreled questions, etc.

Step 2: Adapt to audience

Use of terms used by the survey takers
Other audience characteristics: language, education level, cultural differences, interest level, etc.

Step 3: Reduce burden on survey taker - don’t make them feel bad, stressed or have to interpret what you are saying

Minutes to complete - you want to keep it short - prevent survey fatigue
Nice to know vs. Need to know - Be choosy, obtain the most amount of data in the least amount of time, limit data collection to information you will use, protect privacy and confidentiality - don’t collect identifying information unless you absolutely need it
Be specific & clear - match your audience and what they know and understand - definitions are helpful, eliminate bias, cultural bias, anti-negative bias
Item order - start off with easy questions (demographics) then progress to harder items, keep open-ended/free response questions to a minimum and save for the end
Intro paragraph - a good intro paragraph will help participants decide if they want to take your survey or not. Try to predict what concerns a survey taker would have and address them immediately. Keep it short, casual, and inviting

Include instructions to survey proctors

Design survey form:

Eye strain (font size, bolding of important words, layout, color use, etc.)
Logo and color

Visually appealing
Make good use of space on the page
The survey should not look like it will take too long or require too much thought

Translation

how will participants access those translated surveys?
How will written comments be translated back into English and analyzed

Paper vs. Online

Do you KNOW participants have the ability and capacity to complete an online survey - WiFi/internet connection, device, tech savviness
Survey logic/survey paths

Pilot survey:

Proofread
Critical friend/representative survey taker - Do they understand the survey questions as you intended? How much time does it take to complete survey? How do testers “feel” after having taken the survey?
Go back and forth between survey design and piloting until you’re comfortable with the end result
Get necessary approvals
Give yourself plenty of time to complete these steps - how long depends

Administer survey:

When to distribute to maximize response rate?

at the close of the event with a captive audience - paper exit ticket
Online

collect participants’ email address
How will you track responses and who to send reminders to?

Work around/avoid major events (holidays, assemblies, etc) and conflicting with other data collection efforts (state testing or a competing survey)

Allow for time to complete

Paper - dedicated time in the schedule
Online - as close to the program’s/event’s conclusion as possible while it’s still fresh in their memory

Give survey takers an appropriate amount of time

Incentives - if there is room in the budget and you give yourself the time and staffing you need to deliver on the incentive

REMEMBER - Plan ahead and plan early

Bad survey = bad data
you can only send out the survey once

7 of 13

Proofread, proofread, proofread

Critical friend/ representative survey taker

Get necessary approvals

Lauren

Write survey items:

Step1: develop the questions:

Align to BIG ?s
What are the indicators of that construct?

FOR EXAMPLE: if Big Question—>are teachers implementing program? Then items maybe: 1) how comfortable/confident are you using the curriculum? 2) How many lessons have you taught from the curriculum? 3) How often do you use the curriculum? (every day, weekly, monthly, etc.)

Guard against bias (don’t ask leading questions)
Double-barreled questions, etc.

Step 2: Adapt to audience

Use of terms used by the survey takers
Other audience characteristics: language, education level, cultural differences, interest level, etc.

Step 3: Reduce burden on survey taker - don’t make them feel bad, stressed or have to interpret what you are saying

Minutes to complete - you want to keep it short - prevent survey fatigue
Nice to know vs. Need to know - Be choosy, obtain the most amount of data in the least amount of time, limit data collection to information you will use, protect privacy and confidentiality - don’t collect identifying information unless you absolutely need it
Be specific & clear - match your audience and what they know and understand - definitions are helpful, eliminate bias, cultural bias, anti-negative bias
Item order - start off with easy questions (demographics) then progress to harder items, keep open-ended/free response questions to a minimum and save for the end
Intro paragraph - a good intro paragraph will help participants decide if they want to take your survey or not. Try to predict what concerns a survey taker would have and address them immediately. Keep it short, casual, and inviting

Include instructions to survey proctors

Design survey form:

Eye strain (font size, bolding of important words, layout, color use, etc.)
Logo and color

Visually appealing
Make good use of space on the page
The survey should not look like it will take too long or require too much thought

Translation

how will participants access those translated surveys?
How will written comments be translated back into English and analyzed

Paper vs. Online

Do you KNOW participants have the ability and capacity to complete an online survey - WiFi/internet connection, device, tech savviness
Survey logic/survey paths

Pilot survey:

Proofread
Critical friend/representative survey taker - Do they understand the survey questions as you intended? How much time does it take to complete survey? How do testers “feel” after having taken the survey?
Go back and forth between survey design and piloting until you’re comfortable with the end result
Get necessary approvals
Give yourself plenty of time to complete these steps - how long depends

Administer survey:

When to distribute to maximize response rate?

at the close of the event with a captive audience - paper exit ticket
Online

collect participants’ email address
How will you track responses and who to send reminders to?

Work around/avoid major events (holidays, assemblies, etc) and conflicting with other data collection efforts (state testing or a competing survey)

Allow for time to complete

Paper - dedicated time in the schedule
Online - as close to the program’s/event’s conclusion as possible while it’s still fresh in their memory

Give survey takers an appropriate amount of time

Incentives - if there is room in the budget and you give yourself the time and staffing you need to deliver on the incentive

REMEMBER - Plan ahead and plan early

Bad survey = bad data
you can only send out the survey once

8 of 13

When to distribute?

Allow for time to complete

Incentives?

Lauren

Write survey items:

Step1: develop the questions:

Align to BIG ?s
What are the indicators of that construct?

FOR EXAMPLE: if Big Question—>are teachers implementing program? Then items maybe: 1) how comfortable/confident are you using the curriculum? 2) How many lessons have you taught from the curriculum? 3) How often do you use the curriculum? (every day, weekly, monthly, etc.)

Guard against bias (don’t ask leading questions)
Double-barreled questions, etc.

Step 2: Adapt to audience

Use of terms used by the survey takers
Other audience characteristics: language, education level, cultural differences, interest level, etc.

Step 3: Reduce burden on survey taker - don’t make them feel bad, stressed or have to interpret what you are saying

Minutes to complete - you want to keep it short - prevent survey fatigue
Nice to know vs. Need to know - Be choosy, obtain the most amount of data in the least amount of time, limit data collection to information you will use, protect privacy and confidentiality - don’t collect identifying information unless you absolutely need it
Be specific & clear - match your audience and what they know and understand - definitions are helpful, eliminate bias, cultural bias, anti-negative bias
Item order - start off with easy questions (demographics) then progress to harder items, keep open-ended/free response questions to a minimum and save for the end
Intro paragraph - a good intro paragraph will help participants decide if they want to take your survey or not. Try to predict what concerns a survey taker would have and address them immediately. Keep it short, casual, and inviting

Include instructions to survey proctors

Design survey form:

Eye strain (font size, bolding of important words, layout, color use, etc.)
Logo and color

Visually appealing
Make good use of space on the page
The survey should not look like it will take too long or require too much thought

Translation

how will participants access those translated surveys?
How will written comments be translated back into English and analyzed

Paper vs. Online

Do you KNOW participants have the ability and capacity to complete an online survey - WiFi/internet connection, device, tech savviness
Survey logic/survey paths

Pilot survey:

Proofread
Critical friend/representative survey taker - Do they understand the survey questions as you intended? How much time does it take to complete survey? How do testers “feel” after having taken the survey?
Go back and forth between survey design and piloting until you’re comfortable with the end result
Get necessary approvals
Give yourself plenty of time to complete these steps - how long depends

Administer survey:

When to distribute to maximize response rate?

at the close of the event with a captive audience - paper exit ticket
Online

collect participants’ email address
How will you track responses and who to send reminders to?

Work around/avoid major events (holidays, assemblies, etc) and conflicting with other data collection efforts (state testing or a competing survey)

Allow for time to complete

Paper - dedicated time in the schedule
Online - as close to the program’s/event’s conclusion as possible while it’s still fresh in their memory

Give survey takers an appropriate amount of time

Incentives - if there is room in the budget and you give yourself the time and staffing you need to deliver on the incentive

REMEMBER - Plan ahead and plan early

Bad survey = bad data
you can only send out the survey once

9 of 13

BAD SURVEY=BAD RESULTS

Use Resource: A Catalog of Biases in Questionnaires

10 of 13

Analyze data:

Describe: data by saying how many, who, what percentage?
Synthesize: what’s the point you what to make? Per Big Questions
How does it inform your why? (Conclusions and recommendations)

11 of 13

What was the purpose?

What did you learn?

What would you decide?

Analyze data:

Describe: data by saying how many, who, what percentage?
Synthesize: what’s the point you what to make? Per Big Questions
How does it inform your why? (Conclusions and recommendations)

Distribute reports:

Program leadership and funders
Consider: Survey takers, those impacted by the program or the loss of the program, other key stakeholders?

Lauren

Analyze data:

Summarize responses - frequencies, percents, comparisons, data visualizations

How will you analyze written comments? quotes, themes, sentiments (positive, negative, neutral), stories

Answer the Big ?s - interpret your results

Who completed the survey? Does this represent everyone who came or who we wanted to hear from?
How did participants react to the program/event?
How can teaching and facilitation be improved?
Did we reach intended outcomes?

What to do about unintended outcomes?

What kinds of training or topics should we cover next time?
What did we learn? What are the recommendations?

Distribute reports:

Explain what you found
What is the best way to convey findings? Formal reports vs. informal data summaries

Make decisions/

What recommendations would you make? Use the feedback you receive - it shows that you value their input

12 of 13

Resources

Access today’s presentation and all the other documents we’ve referenced here:

https://tinyurl.com/sl9jeqy

1 of 13

2 of 13

3 of 13

4 of 13

5 of 13

6 of 13

7 of 13

8 of 13

9 of 13

10 of 13

11 of 13

12 of 13

13 of 13