Jiyong Park
Bryan School of Business and Economics
University of North Carolina at Greensboro
Quasi-Experiments
1
Korea Summer Workshop on Causal Inference 2022
Korea Summer Workshop on Causal Inference 2022
Session Website: https://sites.google.com/view/causal-inference2022
Boot Camp for Beginners
Quasi-Experiments
Quasi-Experiments
2
Korea Summer Workshop on Causal Inference 2022
Korea Summer Workshop on Causal Inference 2022
Quasi-Experimental Designs
: Research Designs without Random Assignment
Quasi-Experiments
3
Korea Summer Workshop on Causal Inference 2022
Causal Hierarchy from the Perspective of Potential Outcomes
Level of Causal Inference
Meta-Analysis
Randomized Controlled Trial
Quasi-Experiment
Instrumental Variable
“Designed” Regression/Matching
(based on causal knowledge or theory)
Model-Free Descriptive Statistics (no causal inference)
Exploiting Research Design
Regression/Matching (little causal inference)
Selection on Unobservables Strategies
Selection on Observables Strategies
Exploiting
Random Assignment
Quasi-Experiments
4
Korea Summer Workshop on Causal Inference 2022
Random Assignment is not Always Feasible
Key Identification Assumption
Is the treatment assignment without random assignment as good as random?
How similar is the control group to the treatment group in the absence of the treatment?
Quasi-Experiments
5
Korea Summer Workshop on Causal Inference 2022
Research Design (1) Institution and Economic Growth
Quasi-Experiments
6
Korea Summer Workshop on Causal Inference 2022
Twin cities
Comparable people
Comparable geography
Comparable culture
Research Design (1) Institution and Economic Growth
Nogales, Mexico
Nogales, Arizona
GDP per capita (2019)
USA: $65,297
Mexico: $9,946
Quasi-Experiments
7
Korea Summer Workshop on Causal Inference 2022
Research Design (1) Institution and Economic Growth
Acemoglu, D., Johnson, S. and Robinson, J.A., 2001. The colonial origins of comparative development: An empirical investigation. American Economic Review, 91(5), pp.1369-1401.
Acemoglu, D. and Johnson, S., 2005. Unbundling institutions. Journal of Political Economy, 113(5), pp.949-995.
Quasi-Experiments
8
Korea Summer Workshop on Causal Inference 2022
Research Design (2) Minimum Wage and Employment
Causal effect
= (Employment after the increase in minimum wage in New Jersey) – (Potential employment if the minimum wage were not increased in New Jersey)
Counterfactual
Causal effect estimation
= (Employment after the increase in minimum wage in New Jersey) – (Employment in Pennsylvania after the increase in minimum wage in New Jersey)
Card, D. and Krueger, A.B., 1994. Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania. American Economic Review, 84(4), pp.772-93.
Comparable control group
Quasi-Experiments
9
Korea Summer Workshop on Causal Inference 2022
Research Design (3) 1995 Chicago Heat Wave
Quasi-Experiments
10
Korea Summer Workshop on Causal Inference 2022
Research Design (3) 1995 Chicago Heat Wave
1995년 여름 미국 시카고에 섭씨 41도까지 올라가는 폭염이 찾아왔고 무려 700여 명이 사망했다. 그중에는 노인과 빈곤층, 1인 가구 거주자가 많았다. 이들은 이웃이나 국가의 외면으로 혼자 더위를 견디다 사망한 것으로 드러났다. 정부가 대비책을 아예 마련하지 않았던 것은 아니었지만, 취약계층이 능동적으로 대비책을 활용할 수 있다고 판단한 것이 문제였다.
1999년 시카고에 또다시 폭염이 찾아왔지만, 이번에는 여름철 기상이변에 대한 대비가 잘 돼 있었다. 경찰은 집집마다 방문하여 취약계층의 안전을 확인했다. 시카고시는 냉방센터를 여럿 만들고 냉방센터에 오려는 사람들이 무료로 버스를 이용할 수 있게 했다. 과거의 실수를 반복하지 않으려는 시카고시의 노력으로 사망자 수가 110명으로 줄었다. 여전히 높은 수치지만 99년도의 더위가 95년도와 크게 다르지 않았다는 점을 생각하면 큰 성과라 할 수 있다.
에릭 클라이넨버그는 “1995년도와 1999년도의 차이는 사회적 연결망에 있었다”고 말한다. 비슷한 폭염 상황이라 해도 취약계층의 안부를 수시로 묻고 그들을 챙기려는 노력이 피해를 크게 줄였다.
Quasi-Experiments
11
Korea Summer Workshop on Causal Inference 2022
Research Design (3) 1995 Chicago Heat Wave
1995년 여름 미국 시카고에 섭씨 41도까지 올라가는 폭염이 찾아왔고 무려 700여 명이 사망했다. 그중에는 노인과 빈곤층, 1인 가구 거주자가 많았다. 이들은 이웃이나 국가의 외면으로 혼자 더위를 견디다 사망한 것으로 드러났다. 정부가 대비책을 아예 마련하지 않았던 것은 아니었지만, 취약계층이 능동적으로 대비책을 활용할 수 있다고 판단한 것이 문제였다.
1999년 시카고에 또다시 폭염이 찾아왔지만, 이번에는 여름철 기상이변에 대한 대비가 잘 돼 있었다. 경찰은 집집마다 방문하여 취약계층의 안전을 확인했다. 시카고시는 냉방센터를 여럿 만들고 냉방센터에 오려는 사람들이 무료로 버스를 이용할 수 있게 했다. 과거의 실수를 반복하지 않으려는 시카고시의 노력으로 사망자 수가 110명으로 줄었다. 여전히 높은 수치지만 99년도의 더위가 95년도와 크게 다르지 않았다는 점을 생각하면 큰 성과라 할 수 있다.
에릭 클라이넨버그는 “1995년도와 1999년도의 차이는 사회적 연결망에 있었다”고 말한다. 비슷한 폭염 상황이라 해도 취약계층의 안부를 수시로 묻고 그들을 챙기려는 노력이 피해를 크게 줄였다.
Causal inference aims at designing intervention strategies of inputs (causes)
to improve an output (outcome).
Quasi-Experiments
12
Korea Summer Workshop on Causal Inference 2022
Research Design (3) 1995 Chicago Heat Wave
노스론데일과 사우스론데일의 사례가 대표적이다. 두 지역은 시카고 서쪽에 인접해 있는 마을이다. 그해 여름 두 지역의 기후는 유사했을 게 불문가지다. 독거노인 비율이나 빈곤율도 비슷했다.
하지만 두 지역의 폭염 피해 수준은 크게 달랐다. 노스론데일에서 폭염으로 숨진 사람은 19명(10만명 당 40명)이었다. 반면 사우스론데일에서는 3명(10만명 당 4명)이 사망했다. 희생자 비율이 10배나 차이가 났던 셈이다. 어떻게 이런 차이가 나타났던 걸까.
노스론데일은 공동화(空洞化)된 마을이었다. 60년대 이후 지역경제가 휘청거리자 인구는 급감했고, 약국/식료품점/음식점 같은 상점도 하나둘 사라졌다. 그러면서 범죄율이 치솟았다. 마약상이 활개를 쳤다. 하지만 사우스론데일은 달랐다. 상점가에는 여전히 사람이 넘쳐났다. 범죄율도 낮았다.
저자는 “위험한 지역의 주민들이 고독사로 죽어간 것에는 그들이 좀처럼 집을 떠나지 못하도록 만드는 사회적 환경에서 살았기 때문”이라고 적었다.
Quasi-Experiments
13
Korea Summer Workshop on Causal Inference 2022
Research Design (3) 1995 Chicago Heat Wave
Klinenberg E (2003) Heat wave: A social autopsy of disaster in Chicago: University of Chicago Press
Quasi-Experiments
14
Korea Summer Workshop on Causal Inference 2022
Research Design (3) 1995 Chicago Heat Wave
Klinenberg E (2003) Heat wave: A social autopsy of disaster in Chicago: University of Chicago Press
Quasi-experiment is to take advantage of research designs in which the control groups are comparable to the treatment groups in all aspects, on average,
but the fact that they are treated or not.
Quasi-Experiments
15
Korea Summer Workshop on Causal Inference 2022
Research Design in Marketing Research
Goldfarb, A., Tucker, C. and Wang, Y., 2022. Conducting research in marketing with quasi-experiments. Journal of Marketing, 86(3), pp.1-20.
Quasi-Experiments
16
Korea Summer Workshop on Causal Inference 2022
Research Design Matters
Angrist, J.D. and Pischke, J.S., 2017. Undergraduate econometrics instruction: through our classes, darkly. Journal of Economic Perspectives, 31(2), pp.125-144.
Keele, L., 2015. The Statistics of Causal Inference: A View from Political Methodology. Political Analysis, 23(3), pp.313-335.
“We call this framework the design-based approach to econometrics because the skills and strategies required to use it successfully are related to research design.” (Angrist and Pischke 2017, p. 126)
“The analyst is successful at identifying the causal effect not because of the complex statistical methods that are applied to the data, but due to the effort in developing a design before data are collected.” (Keele 2015, p. 331)
Quasi-Experiments
17
Korea Summer Workshop on Causal Inference 2022
“Taken together, therefore, the Laureates’ contributions have played a central role in establishing the so-called design-based approach in economics. This approach – aimed at emulating a randomized experiment to answer a causal question using observational data – has transformed applied work and improved researchers’ ability to answer causal questions of great importance for economic and social policy using observational data.” (P. 2)
Quasi-Experiments
18
Korea Summer Workshop on Causal Inference 2022
Korea Summer Workshop on Causal Inference 2022
Overview of Quasi-Experimental Methods
Quasi-Experiments
19
Korea Summer Workshop on Causal Inference 2022
Counterfactual Revisited
Counterfactual
| Treatment | Potential Outcomes | Causal Effect | |
Subject i | | | | |
1 | 1 | 3 | counterfactual | ATE on the Treated (ATET) |
2 | 1 | 1 | counterfactual | |
3 | 0 | | 1 | ATE on the Untreated (ATEU) |
4 | 0 | | 1 | |
Counterfactual
ATE
For ATET = ATE, the treatment and control groups should be comparable.
Quasi-Experiments
20
Korea Summer Workshop on Causal Inference 2022
Data Structure from the Perspective of Counterfactual
Time
Treated Unit A
Outcome
Outcome
Outcome
Treatment
Counterfactual
Outcome
Counterfactual
Outcome
Counterfactual
Quasi-Experiments
21
Korea Summer Workshop on Causal Inference 2022
Data Structure from the Perspective of Counterfactual
Time
Treated Unit A
Treated Unit B
Untreated Unit C
Untreated Unit D
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Treatment
Counterfactual
Counterfactual
Counterfactual
Counterfactual
Counterfactual
Counterfactual
Quasi-Experiments
22
Korea Summer Workshop on Causal Inference 2022
Data Structure from the Perspective of Counterfactual
Time
Treated Unit A
Treated Unit B
Untreated Unit C
Untreated Unit D
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Outcome
Treatment
Counterfactual
Counterfactual
Counterfactual
Counterfactual
Counterfactual
Counterfactual
Quasi-Experiments
23
Korea Summer Workshop on Causal Inference 2022
What’s Your Research Design and Data Structure?
Quasi-experimental designs are available.
Treatment and control groups are observed.
Parallel trend assumption is valid.
Treatment is assigned by arbitrary threshold.
There is an exogenous variable that can induce the treatment.
No
Local Average Treatment Effect
Yes
No
DID + Matching
Synthetic Control
Regression Discontinuity
Quasi-Experiments
The goal is causal inference, and random assignment is feasible.
Matching/
Weighting
There is a single treatment group and no information about functional forms.
Regression
Selection on Observables
No
Yes
No
Yes
No
Note that the flowchart may depend on the context.
Yes
Randomized Controlled Trial
No
Longitudinal data is observed.
Interrupted Time-Series Analysis
Yes
No
Longitudinal data is observed.
Not feasible
No
Yes
Yes
Difference-in-Differences (DID)
Yes
or
No
Yes
Control Function & Selection Model
or
Instrumental Variable
Quasi-Experiments
24
Korea Summer Workshop on Causal Inference 2022
What’s Your Research Design and Data Structure?
Treatment and control groups are observed.
Parallel trend assumption is valid.
Treatment is assigned by arbitrary threshold.
There is an exogenous variable that can induce the treatment.
No
Local Average Treatment Effect
Yes
No
DID + Matching
Synthetic Control
Regression Discontinuity
Quasi-Experiments
The goal is causal inference, and random assignment is feasible.
Matching/
Weighting
There is a single treatment group and no information about functional forms.
Regression
Selection on Observables
No
Yes
No
Yes
No
Yes
Randomized Controlled Trial
No
Longitudinal data is observed.
Interrupted Time-Series Analysis
Yes
No
Longitudinal data is observed.
Not feasible
No
Yes
Yes
Difference-in-Differences (DID)
Yes
or
No
Yes
Control Function & Selection Model
or
Instrumental Variable
Quasi-experimental designs are available.
Note that the flowchart may depend on the context.
Quasi-Experiments
25
Korea Summer Workshop on Causal Inference 2022
Difference-in-Differences
| Treatment Group | After Period | Actual Treatment | Potential Outcomes | Causal Effect | |
Subject i | | | | | | |
1 | 1 | 0 | 0 | | 1 | ATET = 1 |
1 | 1 | 1 | 3 | 1 + 0.5 | ||
2 | 1 | 0 | 0 | | 0 | |
1 | 1 | 1 | 1 | 0 + 0.5 | ||
3 | 0 | 0 | 0 | | 1 | |
0 | 1 | 0 | | 1 | ||
4 | 0 | 0 | 0 | | 0 | |
0 | 1 | 0 | | 1 | ||
Average increase by 0.5
Parallel trend assumption is required.
If it is not satisfied, the control group can be recomposed using matching.
Difference-in-Differences (DID)
Quasi-Experiments
26
Korea Summer Workshop on Causal Inference 2022
What’s Your Research Design and Data Structure?
Treatment and control groups are observed.
Parallel trend assumption is valid.
Treatment is assigned by arbitrary threshold.
There is an exogenous variable that can induce the treatment.
No
Local Average Treatment Effect
Yes
No
DID + Matching
Synthetic Control
Regression Discontinuity
Quasi-Experiments
The goal is causal inference, and random assignment is feasible.
Matching/
Weighting
There is a single treatment group and no information about functional forms.
Regression
Selection on Observables
No
Yes
No
Yes
No
Yes
Randomized Controlled Trial
No
Longitudinal data is observed.
Interrupted Time-Series Analysis
Yes
No
Longitudinal data is observed.
Not feasible
No
Yes
Yes
Difference-in-Differences (DID)
Yes
or
No
Yes
Control Function & Selection Model
or
Instrumental Variable
Quasi-experimental designs are available.
Note that the flowchart may depend on the context.
Quasi-Experiments
27
Korea Summer Workshop on Causal Inference 2022
Synthetic Control
| Treatment Group | After Period | Actual Treatment | Potential Outcomes | Causal Effect | |
Subject i | | | | | | |
1 | 1 | 0 | 0 | | 1 | |
1 | 1 | 1 | 3 | | ||
2 | 1 | 0 | 0 | | 0 | |
1 | 1 | 1 | 1 | | ||
3 | 0 | 0 | 0 | | 1 | |
0 | 1 | 0 | | 1 | ||
4 | 0 | 0 | 0 | | 0 | |
0 | 1 | 0 | | 1 | ||
(2) Predicting the counterfactual (a.k.a. synthetic control)
Synthetic Control (SC)
(1) Training a prediction model
Even parallel trends assumption may not be necessary.
Quasi-Experiments
28
Korea Summer Workshop on Causal Inference 2022
Synthetic Control vs. Difference-in-Differences
Difference-in-Differences
Difference-in-Differences + Matching
Quasi-Experiments
29
Korea Summer Workshop on Causal Inference 2022
Synthetic Control vs. Difference-in-Differences
Synthetic Control
Quasi-Experiments
30
Korea Summer Workshop on Causal Inference 2022
What’s Your Research Design and Data Structure?
Treatment and control groups are observed.
Parallel trend assumption is valid.
Treatment is assigned by arbitrary threshold.
There is an exogenous variable that can induce the treatment.
No
Local Average Treatment Effect
Yes
No
DID + Matching
Synthetic Control
Regression Discontinuity
Quasi-Experiments
The goal is causal inference, and random assignment is feasible.
Matching/
Weighting
There is a single treatment group and no information about functional forms.
Regression
Selection on Observables
No
Yes
No
Yes
No
Yes
Randomized Controlled Trial
No
Longitudinal data is observed.
Interrupted Time-Series Analysis
Yes
No
Longitudinal data is observed.
Not feasible
No
Yes
Yes
Difference-in-Differences (DID)
Yes
or
No
Yes
Control Function & Selection Model
or
Instrumental Variable
Quasi-experimental designs are available.
Note that the flowchart may depend on the context.
Quasi-Experiments
31
Korea Summer Workshop on Causal Inference 2022
Interrupted Time-Series Analysis
| Treatment Group | After Period | Actual Treatment | Potential Outcomes | Causal Effect | |
Subject i | | | | | | |
1 | 1 | -1 | 0 | | 1 | ATET = 0 |
1 | 0 | 0 | | 2 | ||
1 | 1 | 1 | 3 | 2 + 1 | ||
Interrupted Time-Series Analysis
+1
Quasi-Experiments
32
Korea Summer Workshop on Causal Inference 2022
Korea Summer Workshop on Causal Inference 2022
Difference-in-Differences
Quasi-Experiments
33
Korea Summer Workshop on Causal Inference 2022
Difference-in-Differences (DID)
Treatment Group
Control Group
Before Treatment
After Treatment
Quasi-Experiments
34
Korea Summer Workshop on Causal Inference 2022
Difference-in-Differences (DID)
Treatment Group
Control Group
Before Treatment
After Treatment
Counterfactual outcome in the absence of treatment
Counterfactual outcome inferred from the control group
Change for the treated in the absence of treatment (not observed)
Inferred from the control group unaffected by the treatment
Inferred counterfactual
Quasi-Experiments
35
Korea Summer Workshop on Causal Inference 2022
Difference-in-Differences (DID)
Treatment Group
Control Group
Before Treatment
After Treatment
Counterfactual outcome in the absence of treatment
Counterfactual outcome inferred from the control group
Change for the treated in the absence of treatment (not observed)
Inferred from the control group unaffected by the treatment
Inferred counterfactual
Quasi-Experiments
36
Korea Summer Workshop on Causal Inference 2022
Difference-in-Differences (DID)
Treatment Group
Control Group
Before Treatment
After Treatment
Counterfactual outcome in the absence of treatment
Counterfactual outcome inferred from the control group
Change for the treated in the absence of treatment (not observed)
Inferred from the control group unaffected by the treatment
Inferred counterfactual
Quasi-Experiments
37
Korea Summer Workshop on Causal Inference 2022
Difference-in-Differences (DID)
Treatment Group
Control Group
Before Treatment
After Treatment
Counterfactual outcome in the absence of treatment
Counterfactual outcome inferred from the control group
Change for the treated in the absence of treatment (not observed)
Inferred from the control group unaffected by the treatment
Inferred counterfactual
Quasi-Experiments
38
Korea Summer Workshop on Causal Inference 2022
Difference-in-Differences (DID)
Treatment Group
Control Group
Before Treatment
After Treatment
Counterfactual outcome in the absence of treatment
Counterfactual outcome inferred from the control group
Change for the treated in the absence of treatment (not observed)
Inferred from the control group unaffected by the treatment
Inferred counterfactual
Quasi-Experiments
39
Korea Summer Workshop on Causal Inference 2022
Identification Assumption for DID
No parallel pretrends
Parallel pretrends
Quasi-Experiments
40
Korea Summer Workshop on Causal Inference 2022
Relative Time Model (Leads-and-Lags Model)
“This is done by creating a second series of time dummies, in addition to the chronological time dummies, which indicate the relative chronological distance between time t and the time Uber is implemented in city j.”
“Econometrically, the primary benefit of this model is that it can determine if a pretreatment trend exists (i.e., a significant difference between treated and untreated counties before treatment) in order to determine if the untreated counties are an acceptable control group. If such a trend exists, it would violate one of the primary assumptions of the model.” (p. 170)
Greenwood, B.N. and Wattal, S., 2017. Show me the way to go home: an empirical investigation of ride-sharing and alcohol related motor vehicle fatalities. MIS Quarterly, 41(1), pp.163-187.
Quasi-Experiments
41
Korea Summer Workshop on Causal Inference 2022
Various Cases of Difference-in-Differences
Danaher, B., Dhanasobhon, S., Smith, M.D. and Telang, R., 2010. Converting Pirates without Cannibalizing Purchasers: The Impact of Digital Distribution on Physical Sales and Internet Piracy. Marketing Science, 29(6), pp.1138-1151.
Quasi-Experiments
42
Korea Summer Workshop on Causal Inference 2022
Various Cases of Difference-in-Differences
Danaher, B., Dhanasobhon, S., Smith, M.D. and Telang, R., 2010. Converting Pirates without Cannibalizing Purchasers: The Impact of Digital Distribution on Physical Sales and Internet Piracy. Marketing Science, 29(6), pp.1138-1151.
Absorbed by unit fixed effects
Absorbed by time fixed effects
Quasi-Experiments
43
Korea Summer Workshop on Causal Inference 2022
Various Cases of Difference-in-Differences
Bell, D.R., Gallino, S. and Moreno, A., 2018. Offline showrooms in omnichannel retail: Demand and operational benefits. Management Science, 64(4), pp.1629-1651.
Offline showroom (no purchase)
30 miles
Control Group
Treatment Group
Warby Parker’s website
Quasi-Experiments
44
Korea Summer Workshop on Causal Inference 2022
Various Cases of Difference-in-Differences
Bell, D.R., Gallino, S. and Moreno, A., 2018. Offline showrooms in omnichannel retail: Demand and operational benefits. Management Science, 64(4), pp.1629-1651.
Offline showroom (no purchase)
30 miles
Control Group
Treatment Group
Warby Parker’s website
Absorbed by unit fixed effects
Quasi-Experiments
45
Korea Summer Workshop on Causal Inference 2022
Various Cases of Difference-in-Differences
Greene, D. and Shenoy, J., 2021. How do anti-discrimination laws affect firm performance and financial policies? Evidence from the post-World War II period. Management Science.
Absorbed by unit fixed effects
Quasi-Experiments
46
Korea Summer Workshop on Causal Inference 2022
Various Cases of Difference-in-Differences
Lins, K.V., Servaes, H. and Tamayo, A., 2017. Social capital, trust, and firm performance: The value of corporate social responsibility during the financial crisis. Journal of Finance, 72(4), pp.1785-1824.
Firms with Higher CSR
Firms with Lower CSR
Before Financial Crisis
After Financial Crisis
Although all firms were under the reach of the financial crisis, this study examines whether firms with higher CSR performed better during the financial crisis.
“We estimate a difference-in-differences model with continuous treatment levels.”
Quasi-Experiments
47
Korea Summer Workshop on Causal Inference 2022
Various Cases of Difference-in-Differences
Brynjolfsson, E., Hui, X. and Liu, M., 2019. Does machine translation affect international trade? Evidence from a large digital platform. Management Science, 65(12), pp.5449-5460.
“We aim to create treatment and control groups using variations in title lengths across different listings… The figure shows variation in title lengths, which we use as the treatment intensity in our continuous DiD estimation”
Quasi-Experiments
48
Korea Summer Workshop on Causal Inference 2022
Various Cases of Difference-in-Differences
Gilje, E.P., 2019. Does local access to finance matter? Evidence from US oil and natural gas shale booms. Management Science, 65(1), pp.1-18.
Access to Local Finance
Shale Boom
(Treatment)
(Natural Shock)
More deposit in local banks by oil firms
Industries with high external financing requirements
Industries with low external financing requirements
Quasi-Experiments
49
Korea Summer Workshop on Causal Inference 2022
Various Cases of Difference-in-Differences
Gilje, E.P., 2019. Does local access to finance matter? Evidence from US oil and natural gas shale booms. Management Science, 65(1), pp.1-18.
Access to Local Finance
Shale Boom
(Treatment)
(Natural Shock)
More deposit in local banks by oil firms
Industries with high external financing requirements
Industries with low external financing requirements
“To achieve this aim, I use a regression form of difference-in-differences, where the first difference (β1) can be thought of as the difference in economic outcomes between boom county-years and nonboom county-years. To identify the effect of the credit component of a boom, I incorporate a second difference (β3), the difference in economic outcomes for industries with high external finance requirements and industries with low external finance requirements.”
Quasi-Experiments
50
Korea Summer Workshop on Causal Inference 2022
Korea Summer Workshop on Causal Inference 2022
Synthetic Control
Quasi-Experiments
51
Korea Summer Workshop on Causal Inference 2022
Synthetic Control vs. Difference-in-Differences
“This method builds on difference-in-differences estimation, but uses systematically more attractive comparisons.”
“The synthetic control approach developed by Abadie, Diamond, and Hainmueller (2010, 2014) and Abadie and Gardeazabal (2003) is arguably the most important innovation in the policy evaluation literature in the last 15 years.” (Athey and Imbens 2017, p. 9)
Athey, S. and Imbens, G.W., 2017. The state of applied econometrics: Causality and policy evaluation. Journal of Economic Perspectives, 31(2), pp.3-32.
Quasi-Experiments
52
Korea Summer Workshop on Causal Inference 2022
Synthetic Control vs. Difference-in-Differences
Quasi-Experiments
53
Korea Summer Workshop on Causal Inference 2022
Synthetic Control vs. Difference-in-Differences
Quasi-Experiments
54
Korea Summer Workshop on Causal Inference 2022
Example (1) Impact of California Anti-Tobacco Legislation
The treated unit (California) is not comparable to untreated units (other states).
Abadie, A., Diamond, A. and Hainmueller, J., 2010. Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105(490), pp.493-505.
Quasi-Experiments
55
Korea Summer Workshop on Causal Inference 2022
Example (1) Impact of California Anti-Tobacco Legislation
Abadie, A., Diamond, A. and Hainmueller, J., 2010. Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105(490), pp.493-505.
Donor Pool
Quasi-Experiments
56
Korea Summer Workshop on Causal Inference 2022
Example (2) Impact of Reunification on West Germany
Abadie, A., 2021. Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), pp.391-425.
Transparency of the counterfactual is one of the most attractive features of the synthetic control.
Donor Pool
Quasi-Experiments
57
Korea Summer Workshop on Causal Inference 2022
How to Construct the Synthetic Control
Abadie, A. and Gardeazabal, J., 2003. The economic costs of conflict: A case study of the Basque Country. American Economic Review, 93(1), pp.113-132.
Abadie, A., Diamond, A. and Hainmueller, J., 2010. Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105(490), pp.493-505.
Predictors for the treated unit
Weighted predictors for the untreated units
There could be multiple predictors that contribute differently to the synthetic control.
Treatment effect for the treated unit after intervention
Counterfactual computed using the synthetic control (weighted control units that best resemble the treated unit)
Quasi-Experiments
58
Korea Summer Workshop on Causal Inference 2022
How to Construct the Synthetic Control
control units
treated unit
pre-intervention outcomes and predictors
(variables in different times are considered separate ones)
Quasi-Experiments
59
Korea Summer Workshop on Causal Inference 2022
How to Construct the Synthetic Control
control units
treated unit
pre-intervention outcomes and predictors
(variables in different times are considered separate ones)
“Computational complexity can be substantially reduced by estimating the composition
of the clusters in a first step (e.g., using K-means)” (p. 11)
Abadie, A. and Zhao, J., 2021. Synthetic controls for experimental design. arXiv preprint arXiv:2108.02196.
Quasi-Experiments
60
Korea Summer Workshop on Causal Inference 2022
How to Construct the Synthetic Control
Doudchenko, N. and Imbens, G.W., 2016. Balancing, regression, difference-in-differences and synthetic control methods: A synthesis (No. w22791). National Bureau of Economic Research.
Quasi-Experiments
61
Korea Summer Workshop on Causal Inference 2022
How to Construct the Synthetic Control
Arkhangelsky, D., Athey, S., Hirshberg, D.A., Imbens, G.W. and Wager, S., 2021. Synthetic difference-in-differences. American Economic Review, 111(12), pp.4088-4118.
Quasi-Experiments
62
Korea Summer Workshop on Causal Inference 2022
How to Construct the Synthetic Control
Kim, S., Lee, C. and Gupta, S., 2020. Bayesian synthetic control methods. Journal of Marketing Research, 57(5), pp.831-852.
Quasi-Experiments
63
Korea Summer Workshop on Causal Inference 2022
How to Construct the Synthetic Control
Basically, the synthetic control approach is the prediction problem.
“Like for the lasso, the goal of synthetic controls is out-of-sample prediction” (Abadie 2021, p. 408)
Abadie, A., 2021. Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), pp.391-425.
Quasi-Experiments
64
Korea Summer Workshop on Causal Inference 2022
Inference for Synthetic Control
Abadie, A., 2021. Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), pp.391-425.
Example (1) Impact of California Anti-Tobacco Legislation
measure of relative difference post and pre
Quasi-Experiments
65
Korea Summer Workshop on Causal Inference 2022
Sensitivity Tests for Synthetic Control
Amjad, M., Shah, D. and Shen, D., 2018. Robust synthetic control. Journal of Machine Learning Research, 19(1), pp.802-852.
Example (2) Impact of Reunification on West Germany
Quasi-Experiments
66
Korea Summer Workshop on Causal Inference 2022
Sensitivity Tests for Synthetic Control
Varian, H.R., 2016. Causal inference in economics and marketing. Proceedings of the National Academy of Sciences, 113(27), pp.7310-7315.
Pre-treatment period
Quasi-Experiments
67
Korea Summer Workshop on Causal Inference 2022
Requirements for Synthetic Control
Abadie, A., 2021. Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), pp.391-425.
Quasi-Experiments
68
Korea Summer Workshop on Causal Inference 2022
Korea Summer Workshop on Causal Inference 2022
Interrupted Time-Series Analysis
Quasi-Experiments
69
Korea Summer Workshop on Causal Inference 2022
What if There is No Control Group?
Quasi-Experiments
70
Korea Summer Workshop on Causal Inference 2022
Example (1) Effect of Bombing on Fertility Rate
Rodgers, J.L., John, C.A.S. and Coleman, R., 2005. Did fertility go up after the Oklahoma City bombing? An analysis of births in metropolitan counties in Oklahoma, 1990–1999. Demography, 42(4), pp.675-692.
“Interrupted time series … gain their advantage from the pretreatment series that allows many potential threats [to validity] to be examined.” (p. 679)
“For Oklahoma County, the broad pattern shows an increase beginning exactly at the “bombing effect” month, nine and a half months after the bombing.” (Rodgers et al. 2005, p. 689)
Quasi-Experiments
71
Korea Summer Workshop on Causal Inference 2022
Example (2) Effect of Online Advertising on Clicks
Brodersen, K.H., Gallusser, F., Koehler, J., Remy, N. and Scott, S.L., 2015. Inferring Causal Impact Using Bayesian Structural Time-Series Models. The Annals of Applied Statistics, 9(1), pp.247-274.
Time series of search-related visits to the advertiser’s website (including both organic and paid clicks).
“Our method generalises the widely used difference-indifferences approach to the time-series setting by explicitly modelling the counterfactual of a time series observed both before and after the intervention… it provides a fully Bayesian time-series estimate for the effect.” (Brodersen et al. 2015)
See CausalImpact R package (http://google.github.io/CausalImpact/)
Quasi-Experiments
72
Korea Summer Workshop on Causal Inference 2022
Korea Summer Workshop on Causal Inference 2022
Wrap-Up
Quasi-Experiments
73
Korea Summer Workshop on Causal Inference 2022
What’s Your Research Design and Data Structure?
Treatment and control groups are observed.
Parallel trend assumption is valid.
Treatment is assigned by arbitrary threshold.
There is an exogenous variable that can induce the treatment.
No
Local Average Treatment Effect
Yes
No
DID + Matching
Synthetic Control
Regression Discontinuity
Quasi-Experiments
The goal is causal inference, and random assignment is feasible.
Matching/
Weighting
There is a single treatment group and no information about functional forms.
Regression
Selection on Observables
No
Yes
No
Yes
No
Yes
Randomized Controlled Trial
No
Longitudinal data is observed.
Interrupted Time-Series Analysis
Yes
No
Longitudinal data is observed.
Not feasible
No
Yes
Yes
Difference-in-Differences (DID)
Yes
or
No
Yes
Control Function & Selection Model
or
Instrumental Variable
Quasi-experimental designs are available.
Note that the flowchart may depend on the context.
End of Document
Quasi-Experiments
74
Korea Summer Workshop on Causal Inference 2022