Metadata

Title: Considerations for Phone Surveys

Purpose: Introduce data quality and management practices and protocols for phone surveys

Learning Goals and Key Takeaways:

Date Created: 11/3/2020

Created by: Rosemarie Sandino

Last Edited on: 11/4/2020

Last Edited by: Rosemarie Sandino

Data Flow for Phone Surveys

Rosemarie Sandino

Technical Products Coordinator

RST, November 2021

Objectives

• Introduce protocols and best practices for data quality and management during CATI interviews
• Introduce basic concepts for conducting CATI interviews
• Answer any questions that you have!

CAPI vs. CATI

CAPI vs. CATI

• CAPI: Computer-Assisted Personal Interviewing
• Enumerator with tablet/phone goes to respondent
• Enumerator reads questions out loud, respondent answers out loud, enumerator enters answer into tablet/phone
• CATI: Computer-Assisted Telephone Interviewing
• Enumerator calls respondent
• Enumerator reads questions over the phone, respondent answers over the phone, enumerator enters answer into tablet/phone

CAPI vs. CATI

CAPI

• Create field plan for teams of enumerators to travel to different areas and track respondents
• Create tracking system: number of attempts, reasons, survey, mop-up
• Large part of enumerator day includes travel, finding respondents, building rapport
• Conduct spot checks and accompaniments

CATI

• Choose between Random Digit Dialing and previously selected sample
• Assign enumerators based on language, gender, or randomly
• Create system for tracking attempts, re-assigning, and reasons for incomplete survey
• Create system for partial submissions
• Enumerator day is in one place making calls

New Considerations for You and Your PI

• Assigning enumerators & re-assigning enumerators
• Tracking # of times each respondent has been reached
• Possible response codes & how to respond
• How the survey works if someone hangs up/call is disconnected before the end of the survey
• No in-person spot checks or accompaniments
• Possible burnout & vicarious trauma for enumerators

​

Assigning Enumerators

• How will you assign enumerators to respondents?
• How will you re-assign enumerators if the language is not a match?
• Can you randomize enumerators and study enumerator effects?
• What will you do if an enumerator leaves the project? What will happen to their assigned respondents?

Respondent Tracking

• How will enumerators keep track of how many people they have contacted?
• How will enumerators keep track of how many times they have contacted a respondent?
• How many complete surveys can you expect a day?
• What will you do if an enumerator leaves the project? What will happen to their assigned respondents?

Response Codes

• What are all possible scenarios that can occur during a phone survey?
• Answer or no answer
• Voicemail, invalid number, rings but no voicemail, etc.
• Answer and hangs up, answer and consents but phone breaks up, complete survey, etc.
• What should an enumerator do at each of these points?
• What are the rates of each code by demographics?

Disconnected Calls

• What should the enumerator do if the call is disconnected in the middle of the survey?
• Can the enumerator move forward and submit a survey if they cannot reach the respondent again?
• Will the respondent have to complete the entire survey again if they can be reached again?
• How should you prioritize module order?

Data Quality

• How can you monitor enumerator performance and data quality without in-person spot checks and accompaniments?
• What is important to monitor?
• What is the capacity of your team to double-check data and review data quality flags?
• What devices are available for making calls and/or collecting data? What is the quality of audio recordings?

Case Management

Case Management

• Default workflow
• Data collection is centered around the form
• Fill Blank Form > Select Form > Start
• Case Management workflow
• Data collection is centered around respondents
• Manage Cases > Select Case > Start assigned form(s)
• Assign respondents to certain users or roles
• Removes issue of tracking sheets and allows for dynamic tracking

Elements of Case Management

Preload dataset with all respondent information

Server holds server dataset and publishes variables to survey

Xlsform pulls variables from server dataset

Collect sends new data back to refresh server dataset

Server Dataset

Server

xlsform

Collect

Journey of a Data Point

Server Dataset

Collect

1

2

Yes

Example forms

We will not be doing an exercise today, but please see these example forms to better understand SurveyCTO CATI workflows:

• SurveyCTO CATI Starter Kit materials
• Use the Basic Sample Form with Case Management
• Re-assignment based on a variable
• Replacement for respondents that can’t be reached
• For more, please see the SurveyCTO CATI Starter Kit Guide and the SurveyCTO Guide to CATI Workflows

SurveyCTO CATI starter kit

SurveyCTO Monitoring Tools

SurveyCTO Monitoring Tools

• Obstacles with CATI Surveys & monitoring:
• No in-person spot checks and accompaniments
• No direct observation of enumerator performance
• Benefits of CATI Surveys:
• No time spent tracking respondents
• All enumerator time is accounted for
• You can use SurveyCTO features to monitor enumerators based on time spent

Data Quality with CATI

Data Quality with CATI Surveys

• Best practice: resolve duplicates before running HFCs
• CATI data has multiple observations for each ID
• Consider breaking your data flow into two parts:
• CATI Progress
• Analyze overall progress, productivity, rates of active/inactive calls, etc.
• Data Quality of Completed Surveys
• Like an in-person survey
• Run the DMS (or your own HFCs)
• Consider rates of refusal, non-response, and drop-off

CATI Survey Progress

• Google integration with SurveyCTO
• Every time a survey is submitted, the variables you have published will be sent to the Google sheet
• Only those marked as publishable!
• Use the data sheet to create new sheets
• Sheets formulas to create summary statistics
• Talk to your PI about what they want to see
• Once it is set up, it automatically refreshes!
• Choose to update by caseid or publish every attempt separately

Google Sheets Integration

Set Up Google Sheets Integration

• Mark all variables that should be used as “yes” in the publishable column
• Do NOT publish PII to a Google sheet
• 4. Export > Advanced: publishing form and dataset data to the cloud
• Find the form > Configure
• Google spreadsheets > Add connection
• Select a Google spreadsheet or choose an existing spreadsheet in Drive
• Select the variables to be published

Set Up Google Sheets Integration

Questions?

Thank you!��researchsupport@poverty-action.org �https://povertyaction.force.com/support