2 of 27

Empirical Tools of Public Finance 3

3.1 The Important Distinction between Correlation and Causality

3.2 Measuring Causation with Data We’d Like to Have: Randomized Trials

3.3 Estimating Causation with Data We Actually Get: Observational Data

3.4 Conclusion

Dan Sacks edited by Achmad Nurdany

P R E P A R E D B Y

2 of 27

3 of 27

Empirical Tools of Public Finance

This chapter focuses on empirical public finance.

Empirical public finance: The use of data and statistical methods to measure the impact of government policy on individuals and markets.
Distinguishing between correlations and causal relationship is the key task in empirical public finance.
Correlated: Two economic variables are correlated if they move together.
Causal: Two economic variables are causally related if the movement of one causes movement of the other.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

3 of 27

4 of 27

The Important Distinction Between Correlation�and Causality

3.1

There are many examples where causation and correlation can get confused.
In statistics, this is called the identification problem: Given that two series are correlated, how do you identify whether one series is causing another?

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

4 of 27

5 of 27

The Problem

3.1

Whenever we see a correlation between A and B, there are three possible explanations:
A is causing B.
B is causing A.
Some third factor is causing both.
The general problem that empirical economists face in trying to distinguish among these three explanations. Correlation alone does not imply causation.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

5 of 27

6 of 27

Example Identification Problem: SAT Prep Courses

3.1

Among Harvard students who took an SAT prep course, SAT scores were 63 points lower than among those who hadn’t.

Do prep courses reduce scores (i.e., A causes B)?
Do low scores cause people to enroll in prep courses?
Does some third factor cause both low scores and enrollment?

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

6 of 27

7 of 27

Randomized Trials as a Solution

Randomized trials solve the identification problem.

Randomized trial: The ideal type of experiment designed to test causality, whereby a group of individuals is randomly divided into a treatment group, which receives the treatment of interest, and a control group, which does not.
Treatment group: The set of individuals who are subject to an intervention being studied.
Control group: The set of individuals comparable to the treatment group who are not subject to the intervention being studied.

3.2

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

7 of 27

8 of 27

Randomized Trials as a Solution

Why do randomized trials solve the problem?

Random assignment rules out reverse causation.
Random assignment means the treatment and control group differ only by treatment. This rules out any third factors causing both treatment and effects.
Any difference between treatment and control group must be due to treatment.
Randomized trials therefore considered the “gold standard” for determining causality.

3.2

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

8 of 27

9 of 27

The Problem of Bias

3.2

The identification problem is a problem of bias.

Bias: Any source of difference between treatment and control groups that is correlated with the treatment but is not due to the treatment.
Randomization eliminates bias, which is why it is the gold standard.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

9 of 27

10 of 27

Measuring Causation with Data We’d Like to Have: Randomized Trials

Randomized trials are useful in medicine and public policy.

ERT: Randomized trials showed that Estrogen replacement therapy raised the risk of heart disease. These trials lead to reduced use of ERT.
TANF: Randomized trials showed that changing welfare programs can encourage employment among recipients.

3.2

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

10 of 27

11 of 27

Why We Need to Go Beyond Randomized Trials

3.2

Even the gold standard of randomized trials have some potential problems.

The results are only valid for the sample of individuals who volunteer to be either treatments or controls—not the population as a whole.
They can suffer from attrition.

Attrition: Reduction in the size of samples over time, which, if not random, can lead to bias estimates.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

11 of 27

12 of 27

Estimating Causation with Data We Actually Get: Observational Data

3.3

Typically, randomized data are not available; researchers rely on observational data.

Observational data: Data generated by individual behavior observed in the real world, not in the context of deliberately designed experiments.

Bias is a pervasive, difficult problem in observation data.
There are, however, methods available that can allow us to approach the gold standard of randomized trials.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

12 of 27

13 of 27

Time Series Analysis: Cash Welfare Guarantee and Hours Worked Among Single Mothers

3.3

Time series analysis: Analysis of two series over time.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

13 of 27

14 of 27

Time Series Analysis: Problems

3.3

One common approach is to use time series analysis.

Time series analysis often produces striking patterns.
Does not separate out causation from correlation.

Different subperiods (1968–1976, 1978–1983, 1993–1998) give different impressions.

Omitted variables may be driving the results—especially the macroeconomy and wage-subsidy programs.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

14 of 27

15 of 27

When Is Time Series Analysis Useful? Cigarette Prices and Youth Smoking

3.3

Sharp, simultaneous changes in prices and smoking rates in 1993 and 1998–onward
Known causes: price war, tobacco settlements

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

15 of 27

16 of 27

Cross-Sectional Regression Analysis

3.3

An alternative to time series analysis is cross-sectional regression analysis.

Cross-sectional regression analysis: Statistical analysis of the relationship between two or more variables exhibited by many individuals at one point in time.
Regression analysis finds the best-fitting linear relationship between two variables.
Regression line: The line that measures the best linear approximation to the relationship between any two variables.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

16 of 27

17 of 27

Cross-Sectional Regression Analysis: Labor Supply and TANF Benefit

3.3

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

17 of 27

18 of 27

Example with Real-World Data: Labor Supply and TANF Benefits

3.3

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

18 of 27

19 of 27

Problems with Cross-Sectional Regression Analysis

3.3

Mothers who receive the largest TANF benefits work the fewest hours.
There are several possible interpretations of this correlation:

Perhaps higher TANF benefits are causing an increase in leisure.
Or perhaps some mothers have a high taste for leisure and wouldn’t work much even if TANF benefits weren’t available—and benefits are low because they aren’t working much.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

19 of 27

20 of 27

Control Variables

3.3

Sometimes, control variables can correct bias.

Control variables: Variables that are included in cross-sectional regression models to account for differences between treatment and control groups that can lead to bias.
If we could measure “taste for leisure,” then we could compare two single mothers with identical taste for leisure, but different TANF benefits.
In reality, probably impossible to measure “taste for leisure” and all other relevant variables.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

20 of 27

21 of 27

Quasi-Experiments

3.3

An alternative approach is to use quasi-experiments.

Quasi-experiments: Changes in the economic environment that create nearly identical treatment and control groups for studying the effect of that environmental change, allowing public finance economists to take advantage of randomization created by external forces.
Policy differences across states and over time often create quasi-experiments.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

21 of 27

22 of 27

3.3

Difference-in-difference estimators are popular quasi-experimental designs.

Difference-in-difference estimator: The difference between the changes in outcomes for the treatment group that experiences an intervention and the control group that does not.
Example: In 1998, Arkansas cut its benefit guarantee from $5,000 to $4,000, but Louisiana did not change policy.

Difference-in-Difference Estimators

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

22 of 27

23 of 27

3.3

Benefits and Labor Supply in Arkansas and Louisiana

Arkansas
	1996	1998	Difference
Benefit guarantee ($)	5,000	4,000	−1,000
Hours worked	1,000	1,200	200
Louisiana
	1996	1998	Difference
Benefit guarantee ($)	5,000	5,000	0
Hours worked	1,050	1,100	50

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

23 of 27

24 of 27

3.3

With quasi-experimental studies, we can never be completely certain that we have purged all bias from the treatment–control comparison.
Quasi-experimental studies use two approaches to try to make the argument that they have obtained a causal estimate.
Intuitive approach: Argue that, given the experiment, most of the bias has been removed.
Statistical: Show that alternative control groups give better results.

Problems with Quasi-Experiments: Bias

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

24 of 27

25 of 27

3.3

Experiments give the reduced form impact of some policy, do not explain why the policy works.
Structural estimates: Estimates of the features that drive individual decisions, such as income and substitution effects or utility parameters.
Reduced form estimates: Measures of the total impact of an independent variable on a dependent variable, without decomposing the source of that behavior response in terms of underlying utility functions.

Problems with Quasi-Experiments: Interpretation

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

25 of 27

26 of 27

Conclusion

3.4

The central issue for any policy question is establishing a causal relationship between the policy in question and the outcome of interest.
How to distinguish causality from correlation, or eliminate bias?
Gold standard: randomized trial.

Alternative methods: time series, cross-sectional analysis, and quasi-experimental analysis.
Each of alternative has weaknesses, but it is possible to overcome the identification problem using careful consideration of the problem at hand.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

26 of 27

27 of 27

Conclusion

3.4

Randomized trials are often infeasible.
Alternative methods include: time series, cross-sectional analysis, and quasi-experimental analysis.
Each of alternative has weaknesses, but it is possible to overcome the identification problem using careful consideration of the problem at hand.

C H A P T E R 3 ■ E M P I R I C A L T O O L S O F P U B L I C F I N A N C E

27 of 27