1 of 42

Fundamentals of Survey Design

September 21, 2026

Public Health in the Rockies

Valeria Ayala Alonso, MPH

Olivia Brandt, MPH

Theresa Mickiewicz, MSPH

Sara Cooper, PhD, MSPH Shannon Pray, MSPH

2 of 42

3 of 42

Today’s Agenda

1

2

Build your own short survey

3

Survey Administration

Survey topic and the questions

4

Survey Structure

4 of 42

Truth is rarely pure and never simple.

Oscar Wilde

Norms: Please speak up, speak out, raise your hand: at times we will provide opportunities for you to take an active role but we encourage your contributions throughout the next 90 minutes. In our rapidly changing world, the approach to survey design must change as well. What was true yesterday, may not be true today. What did work in one population may not work with another. Everytime we come out of the field, we have new revelations, or common narratives or best practices get blown up. SO, we welcome your experience. Let’’s share that knowledge and learn from each other.

We are going to cover a lot of content in the next ~ 90”. And before we get started, I’d like to share some over-arching thoughts I have about survey designs. Survey design is an art: as you seek the truth, you are performing a balancing act where you must weigh best practices, your experience in developing and implementing surveys, knowledge of your population , respect for your respondents. Your resources: How much do you have. Can you provide an incentive? Do you have enough trained interviewers for several days of intercept surveys. You will always be thinking about the pros and cons of different approaches.

5 of 42

Today’s Agenda

1

Build your own short survey

6 of 42

Build your Survey: Guidelines

~ 10 – 12 questions
What is the topic or the focus of your survey?

Consider your scope

Develop your questions or blocks of questions (also called constructs):

Apply skip patterns appropriately

Structure of your survey
Who is your population of interest?
How will they complete the survey?

It will be about 10-12 questions:

Topic or focus of your survey: be clear and concise: think about your scope: do you want to pick a very specific focus and go deep? Or do you want to take a more broad approach and develop blocks of questions?

As you develop the questions, keep them simple as possible and ask yourself: will the answers to your questions provide the information you’re looking for? As you develop the questions, return to your topic or focus. Avoid nice-to-know and again,

The structure of your survey: your survey should flow: again. Blocks help you organize your thoughts and that translates into an easier experience for your respondents as well.

Also consider the population you are surveying: in general, you want to avoid jargon, scientific speak and keep it at a 6^th grade reading level, considering the median reading level of people in the US is 8^th grade.

And how will they complete the survey? This has implications about how you will ask the questions.

7 of 42

Today’s Agenda

2

Survey topic and the questions

8 of 42

Example Survey Topic

Do people in my community eat healthy foods?

Block 1: Are fresh and healthy food options available to community members?

Block 4: Are culturally familiar foods available in the community?

Block 3: What do people in my community eat?

Block 2: What are the community members perceptions of a healthy diet?

9 of 42

Build your Survey: 10 minutes

I would like to understand college students' experience with oral nicotine pouches at a local University.

I would like to know what resources are available for chronic disease management in my community.

I would like to know if providing fresh and local foods in corner stores improves community member diets.

10 of 42

Today’s Agenda

2

Survey topic and the questions

11 of 42

Non-Response Bias

When those who complete the survey are different from those who don’t.

The Questions: Sources of Bias

Response Bias

When respondents are not truthful in their responses.

12 of 42

It is a problem for convenience samples when:

The condition of interest is rare
You seek to engage individuals who:

Live with culturally sensitive issues or behaviors that are often socially stigmatized
Don’t have trust in research or public health institutions
Are generally disinclined to participate, for example, young adults

Non-Response Bias

13 of 42

Screen people in/out and stratify your analyses
Engage and recruit from populations with a high proportion of the condition, exposure or behavior of interest
Use incentives
Assure participants of confidentiality and anonymity
Engage potential participants in the importance of the survey and how the data will be used
Consult experts or the literature for ways to engage populations that may be difficult to engage

Ways to Attenuate Non-Response Bias

14 of 42

Response Biases

Poor question design

Comprehension Bias

Recall Bias

Social Desirability Bias

15 of 42

Poor Question Design

Do you disagree that marijuana makes it harder to think or do things?

Yes
No

In your opinion, do you think marijuana makes it harder to think or do things?

Yes
No

How strongly do you agree or disagree with the following statement: Marijuana use makes it harder to think.

Strongly Agree
……
Strongly Disagree

16 of 42

Has a doctor ever told you that you have nephropathy?

Comprehension

Has a doctor ever told you that you have kidney disease?

Has a doctor ever told you that you have nephropathy? This means that one or more of your kidneys has been damaged and doesn’t work as well as they used to.

17 of 42

When was the last time you were screened for breast cancer?

Days
Weeks
Years

Recall Bias

In the last year, were you screened for breast cancer?

Thinking about the last time you were screened for breast cancer, was it more than 2 years ago or less than 2 years ago?

18 of 42

When is the last time you got black-out drunk?

Days
Weeks
Years
Don’t remember

Social Desirability Bias

How often do you socialize at a bar, brewery or tavern?

When you go out with friends, how often does one or more of your friends drink excessively? By excessively I mean slurred speech, stumbling, etc….

19 of 42

How tall are you (in inches)?

Question Order Bias

How much do you weigh?

How many times per week or per month did you take part in moderate physical activity during the past month?

When you took part in this activity, for how many minutes or hours did you usually keep at it?

20 of 42

Sensitive Questions

Ask yourself "Is the question necessary?"
Explain why you are asking the question
Reinforce anonymity, confidentiality and data destruction policies and procedures
Build up to the question: don’t ask right away
Question-loading or using ‘normalizing’ or ‘forgiving’ language

21 of 42

Sensitive Questions

Embed the question in another question

In the past 12 months, have you done any of the following health-related activities?

	Yes	No
Had a routine medical check-up?	⃝	⃝
Received a vaccination?	⃝	⃝
Been tested for a sexually transmitted infection?	⃝	⃝
Visited a dentist?	⃝	⃝
Sought mental health services?	⃝	⃝

22 of 42

Open vs. Close-ended Questions

Open-ended:

Why did you first smoke a cigarette? _____________

Close-ended:

Why did you first smoke a cigarette?

�

Curious
Peer Pressure
To get a buzz from nicotine
Other: ________________
Don’t know/Not sure
Prefer not to answer

23 of 42

Open vs. Close-ended Questions

Why did you first smoke a cigarette?

�Select all that apply.

Socially, friends/family doing it, sharing
Was at a bar or social gathering where alcohol was being served
Peer pressure
For general enjoyment or pleasure, Ilike it
To try something new or different
To get a high or buzz from nicotine
To help with stress, anxiety, or nervousness
To stay awake or alert
To help with focus
To lose or control weight
Other, specify:
Don't know/Not sure
Prefer not to answer

24 of 42

Is it OK to force an answer?

Necessary for screening questions/eligibility
Avoid requiring all survey questions.
Add "Prefer not to answer" or "Not applicable" options to reduce participant frustration.

First, there are a couple of questions to ensure you

you are still eligible.

Are you currently a resident of Colorado?

Yes
No

25 of 42

When should "IDK" be a choice?

How strongly do you agree or disagree that using nicotine pouches is less harmful to a person's health than using nicotine vaping products?

Strongly agree
Agree
Disagree
Strongly disagree
Don't know/Not sure
Prefer not to answer

26 of 42

Likert Scales

Pros

They are familiar to many people
More sensitive than yes/no
Effective in health and satisfaction surveys
Can collapse categories during analysis

Cons

May be difficult for some populations
Central tendency bias
Acquiescence bias

27 of 42

Likert Scales

Unipolar

How supportive are you of a tax on sugar-sweetened beverages?

Extremely supportive
Very supportive
Slightly supportive
Not at all supportive

Bipolar

To what extent do you support or oppose a tax on sugar-sweetened beverages?

Strongly support
Somewhat support
Somewhat oppose
Strongly oppose

28 of 42

Mutually Exclusive Categories

Which best represents your total household income before taxes during the past 12 months?

Less than $50,000
$50,000 to $100,000
$100,000 to $500,000
$500,000 or greater
Don't know
Prefer not to answer

29 of 42

Additional Thoughts

Consider using previously validated questions when possible
Validate the questions and questionnaire among a population of people who are similar to those you wish to survey
If you suspect that your question is biased or completely off-base, run it through AI

30 of 42

Using AI for question validation

Do you consider yourself to be transgender?

No
Yes, Transgender, male-to-female
Yes, Transgender, female-to-male
Yes, Transgender, gender-nonconforming
Don't know/Not sure
Prefer not to answer

31 of 42

Today’s Agenda

3

Survey Structure

32 of 42

Survey Structure

Introductory Text/Consent
Screening questions
Topic of interest: blocks of questions 1, 2, 3……….
Sensitive questions/Demographics
Contact Information

33 of 42

Introductory Text/Consent

State the purpose of the survey
Express that survey is voluntary
Explain how long survey will take
Describe how you will protect personal information, anonymity and confidentiality
Clarify the incentive

34 of 42

Screening Questions

May need to be mandatory
May need the ‘true’ screening questions embedded in multiple questions to avoid ‘eligibility gaming’

35 of 42

Survey Questions

Constructs or blocks of questions 1, 2, 3……….

Helps you organize your thoughts
Improves survey flow

Front load the most important questions
Create a mock table with fabricated answers

36 of 42

Sensitive Questions/Demographics

If they are not part of the screening or critical for analysis: put them at the end
Use validated questions if possible
Questions such as income are often problematic both in the way the question is asked and the way it is interpreted.

37 of 42

Today’s Agenda

Survey Administration

4

38 of 42

Survey Administration

Considerations

Your population of interest

Where are they?
How do they prefer to communicate?
What is their tech capacity?

Your resources
The complexity or simplicity of your questionnaire

39 of 42

Mode of Contact/Recruitment

Intercept: in person
Passive intercept: flyers, referrals or the local paper
Phone
E-mail
Text
Postal mail

40 of 42

Mode of Completion

In person
Phone
On-line
Text
Postal mail

41 of 42

Final Thoughts

Survey design is an art. And you have tools:

Validated questions and questionnaires
Best practices
Your intuition
Your experience
Artificial Intelligence
Testing, testing, testing

42 of 42

Conclusion

Please let us know how we did! Scan this QR Code:

Resources:

Please see the handout on your table for additional resources (The VAP), Bit.ly/CEPEGhome

Questions?