Best Practices for Developing Surveys

Basics of Sampling Methods

September 11, 2025

Vikram Koundinya, CE Evaluation Specialist & Associate Professor of CE

Kit Alviz, Analyst, UC ANR Program Planning and Evaluation

Samuel Ikendi, Academic Coordinator, Climate Smart Agriculture, UC ANR

Guest Speakers

Jocelyn Mobley, Program Evaluation Coordinator, UC ANR

Christie Hedrick, Statewide Manager, Expanded Food and Nutrition Education Program, UC ANR

Anticipated Outcomes

Participants will have:

1. Understanding of the best practices for designing online and paper surveys
2. Hands on experience developing good survey questions
3. Understanding of online survey bot mitigation and detection methods
4. Understanding of different outcome measurement survey designs
5. Understanding of different survey sampling methods and when to use them

​

​

Agenda

• Best practices for developing surveys
• Guidelines for writing survey questions
• Survey questions activity & group discussion
• Break - 5 min
• Bot detection and mitigation & UC ANR Qualtrics
• Outcome Measurement Survey Designs
• Sampling Methods
• Wrap up & training evaluation

Best Practices for Developing & Administering Surveys

(Dillman, Smyth, and Christian, 2014; Newcomer, Hatry, and Wholey, 2015)

Online Surveys

• Decide how the survey will be programmed and hosted
• Evaluate the technological capabilities and other characteristics of the sample
• See that questions display similarly across different devices and browsers
• Have interesting and informative welcome and closing screens

​

Online Surveys contd…

• Decide how many questions you want per page
• Allow respondents to back up in the survey
• Double check the multiple answer choice questions
• Keep the flow of low to high or negative to positive consistent

Online Surveys contd…

• Consider whether or not to use ‘progress indicator’
• Allow respondents to save and finish the survey later
• Use multiple contacts or follow up and vary the message
• Take steps to ensure that emails are not flagged as spam
• Hit ‘Publish’ button

Improving Response Rate Using Qualtrics

• Not all questions in one page
• Addition of progress bar
• Consistency in answer categories
• Use skip/display logic as needed
• Consider using special features
• A minimum number of open-ended questions
• Allow respondents to report problems and ask questions
• Professional look to the survey

• Pilot testing
• Anonymity or confidentiality?
• Benefits of taking the survey
• Relevancy of the topic
• Value of feedback
• Incentives

​

Mailed Surveys

• Construct in booklet form
• Decide question layout and order
• To the extent possible, personalize contacts
• Create interesting and informative front and back cover pages

​

​

Mailed Surveys contd…

• Take steps to ensure that mailings will not be mistaken for junk mail or marketing materials
• Strategically time all the contacts
• Assign an individual ID number to each sample member
• Send a token of appreciation with the survey request

​

Other Survey Best Practices

• Keep the email/letter short and to the point
• Use smaller paragraphs and sentences
• Paper surveys: Have white space (less busy)
• Look up design/visualization principles
• Keep it short!

Survey Incentives & �Response Rates

​

13

Writing Good Survey Questions

Guidelines

• Questions should be simple, short, and direct.

​

• Phrase in a way that they can be understood by every respondent.

​

• Do not use abbreviations, slang, or acronyms.

​

• Phrase questions so as to elicit unambiguous answers. Quantify responses whenever possible.

14

​

• Avoid bias that may predetermine a respondent’s answer.

​

• Avoid questions that might lead to unstated assumptions.

​

• Avoid leading questions, which imply a desired response.

​

• Avoid questions that may elicit embarrassment, suspicion, or hostility in the respondent.

​

​

​

​

​

​

​

​

​

15

​

• Avoid double-barreled questions.

​

• Avoid random questions.

​

• Avoid double negatives.

​

• Make sure alternatives are exhaustive. Have ”other” option where there are wide range of possibilities.

​

• Have mutually exclusive answer choices.

​

​

​

​

​

​

​

​

​

16

• Make sure the respondents have the information necessary to answer the questions.

​

• Have equal variation on both the sides of the rating scale.

​

• Ensure the question stem matches the answer choices.

​

• Have number of scale points based on the purpose and literature.

17

• Give a timeframe/time range when necessary.

​

• Have an importance prompt and bigger text box for open-ended questions.

​

• Have a question inviting any other comments.

​

• Keep the survey as short as possible.

18

Survey Questions Activity

​

The following list of survey questions are (lightly) modified from a variety of real surveys. Spot the issues and improve the questions based on guidelines presented.

​

​

Group Discussion

5-minute Break

​

Bot (and Fraud) Detection

• Survey Design
• Require at least one open-ended question
• Collect paradata/survey administrative data
• Add domain-knowledge check questions
• Ask zipcode
• Add timestamps to each question page
• Collect personal, verifiable information

Remember: AI Bots learn

​

​

Bot (and Fraud) Detection

• Response Cleaning
• Review open-ended questions
• Inconsistent, unlikely, or contradictory answers
• Consecutive responses (1 min)
• Submission time
• Take out all speeders
• Look for straightling or zigzagging in matrix questions
• Unusual email addresses patterns
• Contact your respondents

Bot (and Fraud) Mitigation

• Use financial incentives with caution
• Administer unique survey link to vetted mailing lists
• Public and social media sharing (QR code, link) with caution
• Separate copy of your survey when sharing publicly
• Use security feature in Qualtrics (but they are not enough)

​

UC ANR Qualtrics Account

UC ANR employees can obtain a UC ANR Qualtrics account at no cost. Highlights:

• Suggested use for Qualtrics includes for workshop evaluations, needs assessments, data collection
• Unlimited surveys but if collecting more than 800,000 survey responses there may be an added fee
• Upload a list of contacts; send smart reminders
• Reports
• Special features: Offline Surveys, File upload, Table of Contents, Bot Detection, Text Message Distribution, API/Integration
• Qualtrics is not a replacement to the ANR Portal Survey Tool and our Qualtrics account does not currently have the option add workshop payment
• Qualtrics offers product trainings and tech support; not ANR IT

Qualtrics SMS Pilot Study

UC ANR’s Nutrition Policy Institute conducted a pilot study to assess if text messaging is an effective tool for disseminating information to CalFresh recipients about a nutrition assistance program they are eligible for.

• one-way SMS distribution capabilities to send a weekly informative text message to study participants, personalized survey links, incomplete survey reminders, and thank you messages.
• two-way SMS distribution capabilities to administer a brief two-question follow-up survey at designated intervals.

Findings

• As of 2025 the feature still seems to be under development. Qualtrics’ documentation needs to be updated. Research an alternative SMS provider.
• Not as intuitive as expected
• Not as reliable as expected

​

• Time intensive to figure out

​

Contact: Samantha Sam-Chen, MPH (she/her/ella)

ssamchen@ucanr.edu

​

​

Outcome Measurement Survey Designs

Adapted from University of Michigan’s My Environmental Education Evaluation Resource Assistant Resource: Types of Evaluation Designs by Michaela Zint

Advantages

​

- Useful when time is an issue or participants are not available before the program/activity begins.

- Measures customer satisfaction and self-reported changes in knowledge, attitude, and intention.

​

​

Disadvantages

​

- There is no comparison data, so it is difficult to determine attribution and the magnitude of outcomes.

​

- Source of bias: social desirability

​

Example: Post-Survey Outcomes Question

CE Advisor Gerry Spinelli’s tip: Post-survey page has link to materials

Advantages

​

- Measures or tests knowledge gain between pre and post.

​

Disadvantages

​

- Requires some thoughtful planning and coordination in order to match pre and post survey responses to an individual for analysis.

- Source of bias: response shift; so do not use self-reported questions.

​

Advantages

- Measures self-reported changes in knowledge, attitude, confidence, and intention with comparison data.

- Useful when a pre survey is not feasible.

- Known to minimize response-shift bias that occurs with traditional Pre/Post

​

Disadvantages

-Source of bias: social desirability, recall, effort justification. Consider pairing with a follow-up to mitigate these biases (see next slide).

​

Example: Retrospective Pre/Post Outcomes Question

Advantages

- Only Follow Up Surveys can measure self-reported behavior change! Can also measure self-reported changes in knowledge and attitude.

- This type of evaluation design is useful when time is an issue or participants are not available after the program/activity ends.

- Can conduct multiple, paired, follow-ups over a longer period of time to assess sustained behavior change, if applicable.

​

​

​

Disadvantages

​

- There is no comparison data, so it is difficult to determine attribution and the magnitude of outcomes.

- Source of bias: recall

​

​

Survey Sampling

Population

• Elements under study (Individuals, schools etc)

Sampling

• Probability vs. Non-probability
• Selection of sample makes study less expensive
• Estimates are sufficiently precise - representative
• Data quality compared to Census

Source: https://www.fynzo.com/wp-content/uploads/2022/09/types-of-sampling-method-1024x681.jpg

Sampling Frame

• Sampling Frame
• From which the individuals /unit are selected
• Things to be considered
• Coverage of the sampling frame
• Individual or Cluster
• Unwanted listing
• Duplication
• Accessibility of sampled elements

​

Deficiency in Sampling Frame

• Missing elements: Some population elements are not included in the sample frame
• Supplement sample frame
• Clusters: Listing refers to group not individuals
• Include all elements in the clusters
• Blanks: Listing in the frame do not relate to elements in the survey population
• Do not include them on the frame
• Duplicate listing: Appears more than one once
• Remove duplicates for the
• Keeping one deleting other

(Lohr, 2009, p. 4)

Source: https://i1.wp.com/conceptshacked.com/wp-content/uploads/2020/12/Probability-Sampling-min.png?resize=1536%2C1024&ssl=1

Simple Random Sampling

• Simplest method
• units equally likely to be selected for the sample
• Replacement/unrestricted or without replacement while selection

​

Systematic Sampling

• Taking every k-th element, starting randomly, with a value of k=N/n.

(N= Population Size; n=Sample Size)

• Choice of k is made based on the estimate of N
• Challenges:
• list is in cyclical arrangement;
• Rounding up or rounding down issues

Stratification

• Population into subpopulations: Stratum
• Selection of elements from Strata
• strata are all presented in the sample
• Proportionate Stratification and Disproportionate Stratification

​

Source: https://www.geeksforgeeks.org/python/stratified-sampling-in-pandas/

Cluster & Multi-stage Sampling

• Samples of clusters are included in the survey
• All elements selected: Cluster Sampling
• Only a sample of elements selected: Two-Stage sampling
• Appropriate if heterogeneity is within the cluster (homogeneity between the clusters)

Source: https://tgmresearch.com/cluster-sampling.html

Non-probability Sampling Designs

Select whoever is present at a given time.

​

• Not usually used

Groups (quotas) are defined by a given characteristic, then individuals that fit those characteristics are selected in any way and placed in those groups.

​

• E.g. Only Men above 50 years

Researchers try to get a sample that is representative or typical of the population.

​

• They need to have prior knowledge of the population.

Researchers randomly identifies the first respondents (seed), who then suggest other people in the same category to include in the study. The sample size grows over time.

• It is used to study difficult to reach populations.

Sample Size Determination

• Depends on several factors
• % of probability
• Margin of error
• Population %

Calculator:

https://select-statistics.co.uk/calculators/sample-size-calculator-population-proportion/

Program Example: Expanded Food and Nutrition Education Program (EFNEP)

Randomization Protocol

• Each county or county cluster is grouped into a sampling plan. Survey Random Sample Calculator is used to determine number of participants to evaluate to achieve a 4-5% margin of error at a 90-95% confidence level.
• Using average class size and enrollment from previous year, the total number of classes to be evaluated is determined and a ratio of 1 out of every X classes is set. Randomization happens by group assuming an average group size.

​

County Example

• 827 adults enrolled. Average group size is 12 individuals per class. Therefore, 20 groups need to be evaluated.
• County educators submit sampling plan form after the first class of the series which gets imported into the county sampling plan spreadsheet. Supervisors oversee sampling spreadsheet and notify educators if their group/class needs to be sampled. Educators survey participants during next class.

​

​

Program Example: Expanded Food and Nutrition Education Program (EFNEP)

Age of participants (separate sampling plans)

• Adults
• Youth: K-2nd, 3rd-5th, 6th-12th

What is measured

• Adults: Pre/post food behaviors through 24-hour diet recalls, adult monthly savings
• Youth: Pre/post food and physical activity knowledge (K-2) or behaviors (3rd-12th)

​

Benefits of Sampling Plan

• Purpose is to retain integrity of the evaluation while reducing the workload on participants and educators
• The sampling ratio changes yearly depending on enrollment numbers from the prior year
• The large size of our program is what makes this a feasible and approved possibility by our funders

​

​

​

Survey Resources

Survey Designs resource (detailed explanation): https://ucanr.edu/sites/CEprogramevaluation/files/294191.docx

​

UC ANR’s Cooperative Extension Program Evaluation website: https://ucanr.edu/sites/CEprogramevaluation/

​

Qualtrics Checklist by Roshan Nayak: https://ucanr.edu/sites/default/files/2024-09/402426.docx

Bibliography

• Ary, D., Jacobs, L. C., Razavieh, A., Sorenson Irvine, C. K., & Walker, D. A. (2019). Introduction to research in education (10th ed). Cengage, 20 Channel Center Street, Boston, MA 20010, USA.
• Colosi, Laura and Dunifon, Rachel. (2006). What’s the difference?: “Post then Pre” & “Pre then Post.” Cornell University Cooperative Extension.
• Dillman, D., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys. The tailored design method (4th ed). John Wiley & Sons, Inc, Hoboken, New Jersey.
• Hill, Laura Griner and Betz, Drew L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation 26(4) 501-517.
• Kish, L. (1965). Survey Sampling. New York: John Wiley.
• Kalton, G. (2020). Introduction to survey sampling (Vol. 35). SAGE Publications, Incorporated.
• Kumar Chaudhary, A., & Israel, G. D. (2016). Influence of importance statements and box size on response rate and response quality of open-ended questions in web/mail mixed-mode surveys. Journal of Rural Social Sciences, 31(3), 140-159.
• Lohr, S. L. (2009). Multiple-frame surveys. In Handbook of statistics (Vol. 29, pp. 71-88). Elsevier.
• Newcomer, K. E., Hatry, H. P., & Wholey, J. S. (2015). Handbook of practical program evaluation. John Wiley & Sons.
• Schaaf, Janel Klatt, John; Boyd, Heather; Taylor-Powell, Ellen. (2005). Program Development and Evaluation. Quick Tips #27 (Using the Retrospective Post-then-Pre Design), #28 (Designing a Retrospective Post-then-Pre Question), #29 When to Use the Retrospective Post-then-Pre Design), #30 (Analysis of Retrospective Post-then-Pre Data). University of Wisconsin, Madison, WI.
• Select Statistical Software. Population Proportion – Sample Size. Retrieved from https://select-statistics.co.uk/calculators/sample-size-calculator-population-proportion/
• Taylor-Powell, E. (1998). Questionnaire design: Asking questions with a purpose. Program Development and Evaluation, University of Wisconsin-Extension.
• University of California, San Diego (2013). Writing good survey questions. Tips & Advice. UCSD Student Research and Information.
• Zint, Michaela. (2009). An introduction to My Environmental Education Evaluation Resource Assistant (MEERA), a web-based resource for self-directed learning about environmental education program evaluation. Evaluation and program planning 09/2009; 33(2):178-9.

​

​

​

Thank You

​

Questions