Metadata

• Title: High Frequency Checks & Back Checks
• Purpose: Introduce and practice IPA’s user-written commands for monitoring incoming data quality
• Learning Goals and Key Takeaways:
• - The role of data quality in IPA’s Minimum Must-Dos and IPA’s user-written commands that achieve those standards
• - Understand format of IPA’s High Frequency Checks and Back Checks commands
• - Experience setting up and interpreting both tools to incorporate in data flow
• Date Created:
• Created by: Christopher Boyer
• Last Edited on: 10/05/2020
• Last Edited by: Rosemarie Sandino

High Frequency Checks & Back Checks

Rosemarie Sandino

Technical Products Coordinator

RST, USA 2021

Data Flow – Importing

Raw Survey Data

(.csv)

Master .do

Import .do

Merge .do

Cleaning .do

Prepped Survey Data

(.dta)

1. Drop unnecessary variables
2. Generate new variables
3. Rename variables
4. Label variables
5. Format variables
6. Only do the cleaning necessary to do checks!

Maintain untouched raw data files!

Data Flow – Monitoring

Enumerator Performance

Prepped Survey Data

(.dta)

High Frequency Checks (HFC)

Back Checks (Audits)

Daily

Checks

Weekly Checks

Spot Checks

-bcstats-

Daily

Report

Weekly Report

BC Diffs

Logical Errors/Discrepancies

Threats to Research Quality

High-Frequency Checks

A high-frequency check is a check of some element of the data collection process, completed on a regular basis as new data comes in. They provide information about:

• The quality of the data
• Threats to research validity
• Enumerator performance
• The quality of the survey
• The data flow (are there systemic flaws?)
• Run checks on your data as often as you can
• Consider running some checks every day, some every few days, some at the beginning of data collection, etc.

​

High Frequency Checks & Back Checks

Reasons for Data Quality Issues

• Enumerator
• doesn’t ask question clearly
• doesn’t probe or make respondent feel comfortable answering the question
• rushes through survey or skips questions
• accidentally types the wrong thing or chooses the wrong answer
• Respondent
• doesn’t understand the question
• doesn’t feel comfortable answering the question
• is tired of answering questions and starts answering randomly
• Survey
• is confusing
• choice lists don’t include the answer the respondent wants to choose
• is too long and cumbersome
• programming error leads to irrelevant question or typo

​

High Frequency Checks & Back Checks

Acting on Data Quality Issues

Consider the reason behind the data quality issue to address it

• What could have caused the issue?
• Can it be fixed? How can it be fixed?
• Who should fix it? What should the response be?
• What is the process for responding?

Remember that no data is perfect

• Prioritize issues you can see and address
• Create a plan for addressing each type of check

​

High Frequency Checks & Back Checks

Data Quality Action Plans

• Discuss with your PI in advance what is most important to monitor
• Create a schedule for running HFCs and sending reports to PIs
• Write out a plan for action steps in advance and how you will address each check you run
• Run check on key variables for outliers.
• For outliers past a certain threshold (that you have created with your PI), give field managers Excel file of flagged values
• Field managers call enumerators with flagged values and ask them to confirm
• Make replacements in replacements file or mark them as “okay” so they don’t show up again

​

High Frequency Checks & Back Checks

HFC Template Workflow

HFC Outputs

Distribute among Field and Research Teams

• HFC Replacements
• Re-training
• Reminders
• Incentives
• Praise
• Survey programming changes

Data Collection

Back checks (MMD for IPA)

• Re-ask within 1-3 days
• Think about types and variance
• Back check > 10% of surveys
• Stratify random assignment by surveyor
• Quickly compare and take action using bcstats
• Backcheck forms should be included in IRB submissions

​

​

​

High Frequency Checks & Back Checks

bcstats to the Rescue!!

• What is it?
• bcstats is an IPA user-written Stata command to compare back check data to survey data and summarize results.
• Syntax:

​

​

​

bcstats, surveydata(filename) bcdata(filename) ///

id(varlist) [options]

Linking the two

Why we do back checks in the first place:

• Identify survey falsifications (type 1)
• Evaluate enumerator performance (type 2)
• Identify enumerator effects or other sources of bias (HFCs/type3)
• Evaluate the reliability of our measures (type 3)

​

Data Flow

Run master do file to:

• Import data
• Merge and/or Prep Data
• Creates: prepped dta file
• Run High-Frequency Checks
• Creates: HFC outputs, tracking data, enumerator performance
• Make replacements
• Creates: adjusted dta file
• Run back check analysis
• Optional: prep back check data as well

​

​

​

​

​

Data Management System

• Consolidate high frequency checks, tracking, and back checks into the same system.
• Create your folder structure/data flow with readme files
• Input and output sheets in Excel that allow you to turn on/off checks, set key variables for checks, formatted summaries

​

Relative Working Directories

• Relative working directories reference your existing working directory
• ../
• Move back one folder

​

​

Data Flow

Minimum Checks

Minimum Checks

Minimum Checks

Additional Checks

Data Import Workflow

1. Download .csv using SurveyCTO Sync

3. Stata Data Set

2. SurveyCTO do-file

HFC Template Workflow

1. HFC Inputs

2. Run Simple Do-file on Data

3. HFC Output Lists

HFC Template Workflow

1. HFC Outputs

3. HFC Replacements

2. Distribute among Field and Research Teams

• Re-training
• Reminders
• Incentives
• Praise
• Survey programming changes

HFC Template Workflow

1. HFC Inputs

3. HFC Output Lists

4. HFC Replacements

2. Run Simple Do-file on Data

File Overview: Input

HFC Input

File Overview: Output

HFC Output

File Overview: Replacement

• A new instructions sheet that explains how to fill out replacements
• Option to drop, replace, or “okay” observations
• Automatically run in DMS

Dashboards

Excel Output - 1. Differences File

The basic idea is to harmonize Excel output with the HFCs

Excel Output - 1. Differences File

The first section identifies the id, survey enum, and back checker

Excel Output - 1. Differences File

The second section identifies the type (1, 2, or 3) and the variable name and label (question text)

Excel Output - 1. Differences File

The third section shows the survey value and the corresponding back check value. Users can also specify additional variables to display (e.g. date)

Excel Output - 2. Error rates by variable

The second tab shows the error rates by variable (i.e. the number of differences divided by the total number of responses) sorted from highest to lowest rate of errors.

Excel Output - 3. Error rates by enumerator

The third tab shows the error rates by enumerator sorted from highest to lowest.

Excel Output - 4. Error rates by back checker

The fourth tab shows the error rates by back checker sorted from highest to lowest.

Excel Output - 5. Statistical test of stability

The fifth tab shows the results of a two-sample mean comparison test of stability for continuous type 2 and type 3 variables.

A p-value below 0.05 suggests that back check data are significantly different than survey data (which could mean improper surveying is skewing the results)

Excel Output - 6. Statistical test of stability

The sixth tab shows the results of a two-sample mean comparison test of stability for binary type 2 and type 3 variables.

Excel Output - 7. Reliability of survey measures

The seventh tab shows the simple response variance and the reliability rato.

• High Frequency Checks Manual is on the Global Help Desk!

• Back Check Manual is on the Global Help Desk!

HFCs Exercise (60 min.)

If you already have ipacheck and bcstats installed:

​

Open Stata, set your working directory, and type the code:

​

ipacheck new, exercise

​

Open exercise_instructions.pdf and begin!

​

​

​

If you have not installed the commands, go to installation instructions on website

Thank you

poverty-action.org

povertyactionlab.org

Thank you

poverty-action.org

Credits

• This presentation was created by Christopher Boyer and references the user-written commands ipacheck (created by Christopher Boyer) and bcstats (created by Matthew White).

​

References

• Addae, Kai. 2017. “Back Check Manual.” Innovations for Poverty Action Global Help Desk. Last modified December 19, 2017. https://povertyaction.force.com/support/s/article/back-check-manual.
• Innovations for Poverty Action. 2017. “high-frequency-checks”. Github repository. https://github.com/PovertyAction/high-frequency-checks.
• Innovations for Poverty Action. 2017. “bcstats”. Github repository. https://github.com/PovertyAction/bcstats.
• Sandino, Rosemarie. 2018. “High Frequency Checks Manual.” Innovations for Poverty Action Global Help Desk. Last modified June 25, 2018. https://povertyaction.force.com/support/s/article/hfcs-manual.

​

​

​

Measurement Theory

var(ε)

Let’s be honest with ourselves when answering a question, respondents are likely drawing their answer at random from some distribution of responses about the true value.

Measurement Theory

var(ε)

For example, if you ask me multiple times “how many times have you washed your hands in the last day” I’m likely to give a slightly different answer depending on my memory, stress level, etc.

Measurement Theory

var(ε)

Across an entire survey this variability in interview responses will tend to increase the variance of an indicator… and possibly decrease our power for detecting significant effects.

var(y)

y_ij = t_ij + ε_i

Measurement Theory

var(ε)

Under certain assumptions (i.e. the survey and back check responses are independent), it can be shown that variance of the measurement error is equivalent to the simple response variance (SRV).

SRV = var(y_i2- y_i1) = var(ε_i)

Measurement Theory

The SRV, by itself, is not a very useful tool for assessing the reliability of different questions. However, it can be used to calculate the reliability ratio

​

SRV = var(y_i2- y_i1)

​

​

​

Reliability Ratio = 1 - SRV / var(y_i)

Measurement Theory

A reliability ratio of greater than 0.8 is considered “high”. The UN recommends a reliability ratio of at least 0.5 in all important survey outcomes.

Measurement Theory

However, to measure the reliability ratio effectively we must take care to ensure that the conditions of the interview in the survey and the back check are as close to the same as possible.

Measurement Theory

A note of caution: there is another source of error, response bias, which is a systematic difference in successive responses.

bias

Measurement Theory

For example, suppose I am very conscious of what FEMALE interviewers might think of me if I reported my true hand washing habits, so then I systematically over report hand washing when the interviewer is female.

bias

H₀: μ₁ = μ₂

H₁: μ₁ ≠ μ₂

Measurement Theory

Unfortunately, the only way to identify this type of error is to either (1) directly measure the “true value” or (2) change the conditions under which the back check is done.

bias

H₀: μ₁ = μ₂

H₁: μ₁ ≠ μ₂

Summary

• The reliability ratio and SRV give an idea of how much variability is due to instability in your instrument, but estimating them requires conditions of survey and back check to match.

​

• On the other hand, stability checks can give you a sense of potential response bias, but estimating them requires knowing the true value or changing the conditions of the back check.

​

​