Lecture 16: Causal inference
case studies II
Sharad Goel
Stanford University
Estimating causal effects from observational data
Response surface modeling�Propensity score matching�Interrupted time series�Difference-in-differences
Optimizing bail decisions�Jung, Concannon, Shroff, Goel, Goldstein, 2017
Judges determine whom to release and whom to detain pending trial, balancing flight risk against the burden of bail on individuals.
Goal is to design more equitable release policies.
The data
150,000 cases for non-violent offenses from a large urban prosecutor’s office.
For each instance, we know the judge’s decision [release or “detain”], the outcome [appear or fail-to-appear], and detailed case information.
Average treatment effect
What is the effect of treatment [ being detained ] on failure to appear rates?
Average treatment effect
What is the effect of treatment [ being detained ] on failure to appear rates?
FTA rate is 9% for those who are detained, and 15% for those who are released.
Average treatment effect
What is the effect of treatment [ being detained ] on failure to appear rates?
FTA rate is 9% for those who are detained, and 15% for those who are released.
Selection bias�Those defendants who were detained are different than those who were released. [ Detained defendants are likely riskier. ]
Response surface modeling
Estimate potential outcomes by fitting a model to predict responses as a function of covariates and treatment.
Response surface modeling
Y is the outcome [ appear or fail to appear ]�X is the vector of covariates [ case details, etc. ]�Z is the treatment [ detain or release ]�
Response surface modeling
Y is the outcome [ appear or fail to appear ]�X is the vector of covariates [ case details, etc. ]�Z is the treatment [ detain or release ]�
Response surface modeling�The ignorability assumption
The ignorability assumption means that given the observed covariates X, the potential outcomes are independent of the treatment Z.
In this case, the treatment is ignorable and we don’t have to worry about selection bias. �[ We revisit this assumption shortly. ]
Response surface modeling�Estimating f
Model for Z=0 is fit on all cases in which the judge released the defendant.�
Model for Z=1 is fit on all cases in which the judge detained the defendant.�
Response surface modeling�Estimating f via more complex models
We can use more complex ML models [ regularized regression, random forest, etc. ] to estimate outcomes, but the basic strategy is the same.
Response surface modeling�Result
On the subset of people who were in reality detained, use the model to estimate Yi(released).
Response surface modeling�Result
On the subset of people who were in reality detained, use the model to estimate Yi(released).
On this subset, the counterfactual FTA rate is 17%.�[ On the set of people released, observed FTA is 15%. ]
Response surface modeling�Result
On the subset of people who were in reality detained, use the model to estimate Yi(released).
On this subset, the counterfactual FTA rate is 17%.�[ On the set of people released, observed FTA is 15%. ]
Average treatment effect on the treated [ ATT ] is�9% - 17% = -8%
Response surface modeling�Result
On the subset of people who were in reality released, use the model to estimate Yi(detained).
Using the model and the observed outcomes, we have estimates of both Yi(detained) and Yi(released) for each defendant, and can then compute the ATE.
Matching
Instead of modeling the response surface, one can attempt to find similarly situated individuals.
Matching
Instead of modeling the response surface, one can attempt to find similarly situated individuals.
For each person detained, find a similar defendant who was released. Use this matched sample to estimate the unobserved potential outcomes.
Can think of treatment as randomly assigned in the matched sample.
Matching
How to find “similar” individuals?
Matching
How to find “similar” individuals?
Exact matching�Find individuals who share exact same observed traits.�[ age, sex, race, criminal history, … ]
Matching
How to find “similar” individuals?
Exact matching�Find individuals who share exact same observed traits.�[ age, sex, race, criminal history, … ]
Exact matching is typically impossible in high dimensions.
Propensity score matching
For each individual, estimate their likelihood of being treated based on all observed traits.�[ Estimate these propensity scores via a big model. ]
Match individuals on their propensities to be treated.
Propensity score matching�Full population
Propensity score matching�Matched population
Propensity score matching�Results
In the sample of released individuals matched to defendants who were detained, FTA rate is 18%.�[ On the set of people released, observed FTA is 15%. ]�[ The response surface model yields an FTA rate of 17%. ]
Designing optimal policies�Beyond average treatment effects
Policy design strategy
Policy design strategy
Evaluating detention policies
Number detained vs. number who fail to appear
Use the response surface model to estimate counterfactual when necessary.
If the policy suggests releasing an individual who was in reality detained, or detaining an individual who was in reality released, the model yields estimates of the unobserved potential outcomes.
Evaluating detention policies
30%
fewer defendants can be detained �without increasing overall flight risk.
Violating ignorability�
Suppose we only observe a defendant’s age X, but judges observe many other factors.
Then, given two defendants of the same age, exactly one of whom is released, the released defendant is likely less risky than the detained one, violating ignorability.
Violating ignorability
When ignorability is violated, then we can’t estimate Y(t) in terms of Pr(Y=1 | X, Z=t).
Violating ignorability
Ignorability is an assumption, and cannot be directly verified.
But we can examine the sensitivity of our conclusions to violations of ignorability.
Sensitivity analysis
Make assumptions about the magnitude and effect of selection, and then examine how estimates change.�[ Many different ways to do this. ]
Sensitivity analysis�Rosenbaum-Rubin sensitivity analysis
Assume there is a binary covariate u that affects both a judge’s decision [ detain or release ] and the outcome [ appear or fail to appear ] conditional on the decision.
Sensitivity analysis�Rosenbaum-Rubin sensitivity analysis
Assume there is a binary covariate u that affects both a judge’s decision [ detain or release ] and the outcome [ appear or fail to appear ] conditional on the decision.
[ New ] ignorability assumption
Sensitivity analysis�Rosenbaum-Rubin sensitivity analysis
Four parameters:
Sensitivity analysis�Rosenbaum-Rubin sensitivity analysis
Decision-making
Risk assessment tools estimate risk. They don’t dictate policy.
Rather than detaining individuals deemed “high” risk of flight, one might attempt to improve appearance rates via automated court reminders or by providing transportation.�[ We’re rolling out an open-source platform to do this. ]