Week 10:

Quasi-Experimental Designs

Program Evaluation - PADP 8640 - Spring 2016

Outline

• Bias and Comparison Groups
• Natural Experiments
• Difference-in-Differences
• Regression Discontinuity Designs

How do we know a program “works”?

• Still getting at the question of how can we accurately measure whether or not a policy achieved anything

​

• Since most policies and interventions are not randomly assigned, we keep running into issues of bias…

​

How do we know a program “works”: Selection Bias

• Selection bias: some individuals more likely to be part of a policy or program than others

​

Results potentially drive by WHO is in

which group, not by WHICH group

someone is in.

​

Internet is the king of selection bias

• Online polls
• Dads on vacation

​

​

​

How do we know a program “works”: Sampling bias

• Are certain types of people more or less likely to be included in our sample?
• If so, study is not externally valid

Examples:

• using NBA/WNBA players to estimate average height of US population
• Comparing graduates of an elite college to all other college graduates

​

All about comparisons...

Selection bias and sampling bias largely about our ability to compare groups

​

• Are treatment and control group equal in expectation?
• Are treatment and control group even relevant to one another?
• Bias means that we cannot account for the way(s) in which treatment and control groups are different

Quasi-Experimental Designs

• Today, we are primarily focusing on attempts to identify and/or make a suitable control group
• This can be a really complex exercise
• What would be a good control group for:
• Economic impact of Braves moving to Cobb County?
• Assessing whether a Starbucks gold card is good for your health?
• The economic benefit of an MPA degree?

Natural Experiments

• What is the appeal of an experiment for evaluation?
• If we cannot manipulate/assign treatment, what features of program or of policy might help approximate an experiment?

Natural Experiments

• In a “natural experiment,” treatment…
• Is exogenous
• Is random forced
• Creates groups that are “equal in expectation”
• What does this mean?

Natural Experiments

• In a “natural experiment,” treatment…
• Is exogenous
• Is random forced
• Creates groups that are “equal in expectation”

​

Geography

Time

Participation Threshold

Example: Students on one side or another of attendance boundary

Example: Changes in benefit program over time

Example: Eligibility Cutoff

Natural Experiments

• The basic premise of a “natural experiment” is that some external factor has addressed some concern about self-selection

Natural Experiments

• Potential concerns:
• Secular trends
• Participant actions (how “arguably exogenous” is treatment?)
• Bandwith choices

Difference-in-Differences

• In “diff-in-diff” design, an exogenous treatment creates four distinct groups

Difference-in-Differences: Example

• Dynarski (2003)
• Social Security Student Benefit
• Program: Cash payment to students with deceased fathers for college tuition support
• Exogenous intervention: Congress abruptly ended the program in 1981

Difference-in-Differences: Example

• Exogenous intervention: Congress abruptly ended the program in 1981

Difference-in-Differences: Example

• Exogenous intervention: Congress abruptly ended the program in 1981

First Difference

Difference-in-Differences: Example

• Exogenous intervention: Congress abruptly ended the program in 1981

Second Difference

Difference-in-Differences: Example

First difference - second difference = difference of differences

Difference-in-Differences: Example

First difference - second difference = difference of differences

Difference-in-Differences: Example

Difference-in-Differences: Good News!

• Difference-in-differences model uses simple regression strategy
• From social security benefits example:

COLL_i = β₀ + Β₁BEFORE_i + β₂FATHERDEC_i + β₃(BEFORE_i x FATHERDEC_i) + ε_i

​

Difference-in-Differences: Good News!

• Difference-in-differences model uses simple regression strategy
• From social security benefits example:

COLL_i = β₀ + Β₁BEFORE_i + β₂FATHERDEC_i + β₃(BEFORE_i x FATHERDEC_i) + ε_i

​

Difference-in-Differences: Good News!

• Difference-in-differences model uses simple regression strategy
• From social security benefits example:

COLL_i = β₀ + Β₁BEFORE_i + β₂FATHERDEC_i + β₃(BEFORE_i x FATHERDEC_i) + ε_i

​

Difference-in-Differences: Example Results

COLL_i = β₀ + Β₁BEFORE_i + β₂FATHERDEC_i + β₃(BEFORE_i x FATHERDEC_i) + ε_i

Difference-in-Differences: Example Results

What is first difference? → y_{before, non-deceased} - y_{after, non-deceased}

What is second difference? → y_{before, deceased} - y_{before, non-deceased}

Difference-in-difference? → [y_{before, deceased}- y_{after, deceased}] - [y_{before, non-deceased} - y_{after, non-deceased]}

Difference-in-Differences (without natural experiment)

Difference-in-differences for interrupted time series data

• While natural experiments are ideal, a diff-in-diff design can be applied to any program change with longitudinal data
• Diff 1: Pre/post program change for program participants
• Diff 2: Pre/post observations for non-participants (to capture secular trend)

​

​

• Often, you’ll also want to include observable covariates.

​

Group Exercise

Group Exercise instructions

Regression Discontinuity Designs

• Many programs and policies have eligibility cutoffs
• Poverty related programs have income cutoffs
• Eligibility for tax credits and expenditures
• Test scores
• Other ideas?

​

• Are people on one side of cutoff different than the other?
• When? Why?

Regression Discontinuity Designs

• Regression discontinuity designs use the arbitrariness of treatment cutoffs to assess causality
• Basic assumption: people close to cutoff on either side are similar
• Two main requirements
• Continuous eligibility index
• Clearly defined cutoff score
• (sub-requirement: cutoff is enforcemed, measurements are not gamed)

Regression Discontinuity Designs

• How would we fit a regression for this?

​

Regression Discontinuity Designs

• LATE (Local Average Treatment Effect) instead of ATE (Average Treatment Effect)
• Why?
• How could you test robustness of your result?

​

General RDD framework

• f(I-I_c)_i is distance from cutoff; in theory, this “soaks up” selection bias
• B₂Y_it-1 is observation from prior time period; adjust for initial differences amongst individuals

Regression Discontinuity Designs

Look for RD designs when policy/program eligibility occurs on a continuous variable:

• Income: what’s the difference between $9,995 and $10,005?
• Distance: 9 miles versus 11 miles?

​

Think substantively: does cutoff carry meaning?

​

Inappropriate for non-continuous cutoffs:

• No spectrum for single moms, college graduates, or being homeless

​

Regression Discontinuity Designs

What did your readings say about RDDs?

​

Why?

Quasi-Experimental Designs

We always ask:

• How close is close enough?
• How similar is similar enough?
• How exogenous is exogenous enough?

Quasi-Experimental vs. Experimental

Experiments aren’t perfect either

​

• Why might an experiment be inappropriate?

​

• What are some potential benefits of an observational study?

​

Questions

Exercise

Group 1: Chairman Christopher Hart

National Transportation Safety Board

​

Group 2: JT Griffin

Senior VP of Pub. Policy, MADD

​

Group 3: Jana Simpler

Director, Governors’ Highway Safety Association

​

Group 4: Rob Tod

Chair, National Brewers’ Association

​

Gwen Ifill!