1 of 21

Using Experiments in Economics

Roman Sheremeta, Ph.D.

Professor, Weatherhead School of Management

Case Western Reserve University

3 of 21

What is economics?�

Economics: the study of choices that economic agents make as they cope with scarcity and the incentives that influence those choices

4 of 21

What is behavioral economics?�

Behavioral economics: the study of choices actually made by economic agents and the corresponding psychological insights leading to such choices

5 of 21

Methods of economic research�

Methods of economic research:

Theoretical method
Econometric/statistical method
Experimental method

Experimental method:

Hypothesis
Experimental design (randomization, treatments, procedures, laboratory or field)
Data analysis (non-parametric tests, regressions)

6 of 21

Using Experiments in Economics

7 of 21

Correlation versus causation�

If you find X and Y are correlated, what can you conclude?

Correlation does not imply causation!

Example 1: Eating breakfast is correlated with success in school for elementary school children

Hypothesis 1: Eating breakfast improves cognition
Hypothesis 2: Not having breakfast is an indication that a kid comes from a bad family: less educated parents, worse socio-economic status, less focus on school at home, lower expectations, etc.

Example 2: For each year of education, a person in the United States earns approximately $4,000 more per year

Hypothesis 1: Education leads to higher earnings
Hypothesis 2: Smarter people choose to go to university, and smarter people will be able to find better jobs

8 of 21

Inferring causality�

What are the common problems when inferring causality?

Measurement error
Omitted variables (Example: ice cream versus drowning)
Simultaneity (more police → less crime, more crime → more police)
Selection (individuals are not assigned randomly into treatments)

How to infer causality?

Multiple regression analysis (trying to control for all relevant factors)
Instrumental variables (trying to find an “instrument”)
Regression discontinuity
Differences-in-differences
Selection model (Heckman two-stage procedure)
Controlled experiment with random assignment is the Golden Standard

9 of 21

Controlled economic experiments�

Economists who have contributed to developing experimental methodology:

Vernon Smith (Nobel Prize in Economics 2002)
Richard Thaler (Nobel Prize in Economics 2017)
John List (Nobel Prize in Economics 20??)

10 of 21

Experimental economics�

Experimental method: Learning from observing behavior in an environment created or modified by the researcher for that very purpose

Why do we use experiments in economics (Smith 1994):

To test economic theories (e.g., weed out bad theories)
To uncover behavioral regularities (e.g., other-regarding preferences, loss aversion, hyperbolic discounting)
To guide new theories (e.g., prospect theory)
To evaluate policy proposals (e.g., schooling curricula, microfinancing, emissions trading)
To design markets (e.g., matching markets for medical doctors, British spectrum auctions, markets for electric power)

11 of 21

Principles of economic experiments�

Main principles (Hertwig and Ortmann 2001):

Random assignment.
Rules are known to the subjects.
Repeated trials.
Financial incentives.
No deception.

12 of 21

Example: cooperation and punishment�

Fehr and Gachter (2000) conducted a laboratory experiment by randomly assigning subjects (students) to either the “no punishment“ or the “punishment“ condition.

In the no-punishment condition: subjects were split into groups of four and each subject had to decide how many of their 20 “points” to contribute to the “common good.” Contributing 1 point yielded 0.4 point to each member of the group.
In the punishment condition: subjects also contributed to the “common good,” but after they observed partners’ contributions, they could punish them (costly) for not contributing.

Main research question:

Does the opportunity for costly punishment increase cooperation?

13 of 21

Example: cooperation and punishment�

1. Random assignment.

Subjects were randomly assigned to different conditions/treatments.

2. Rules are known to the subjects.

Subjects knew the rules of the game.

3. Repeated trials.

Each condition/treatment lasted for 10 periods.

4. Financial incentives.

Experimental “points” were converted to cash.

5. No deception.

There was no deception in the experiment.

14 of 21

Example: cooperation and punishment�

Results of the experiment by Fehr and Gachter (2000):

The opportunity for costly punishment increases cooperation.

What can we make of these results?

All studies are judged by at least two criteria: internal validity and external validity.

15 of 21

Internal validity�

Internal validity: the criterion that a researcher’s conclusions about what happened in the study are well-grounded.

There is a statistical basis for conclusion.
There are no alternative explanations.

The study of Fehr and Gachter (2000) claiming that punishment increases cooperation is internally valid because:

There is a statistical basis for concluding that there is a positive relationship between opportunities for punishment and cooperation.
There are no alternative explanations other than the causal one.

16 of 21

External validity�

External validity: The criterion that a study's conclusions be relevant for other situations.

The sample being studied is similar to the rest of the population.
The situation created in the experiment is not different from a typical relevant situation (“ecological validity”).
Subjects do not know that they are being studied (“demand effect”).

The study of Fehr and Gachter (2000) claiming that punishment increases cooperation has weak external validity because:

Students do not represent all people.
The situation is an imperfect representation of real institutions.
There could be a demand effect to punish.

17 of 21

Internal versus external validity�

There is a tradeoff between internal and external validity:

Experiments have high internal validity since they strip away aspects of the environment that might lead to alternative explanations.
But precisely because experiments strip away much of the richness of a real-life situation, experiments may have lower external validity.

One solution to the problem of external validity is field experiments (List 2011).

18 of 21

Example: recognition and giving�

Samek and Sheremeta (2017) conducted a field experiment by randomly assigning 205 parents who came to pick up their children to four conditions:

No recognition: all individuals could donate to the Red Cross and donation amounts were publicly displayed
Full recognition: all individuals’ names publicly displayed next to their donations
Positive recognition: the names of the highest two givers publicly displayed next to their donations
Negative recognition: the names of the lowest two givers (or non-givers) publicly displayed

Main research question:

What kind of recognition is more effective at increasing giving?

19 of 21

Example: recognition and giving�

Results of the experiment by Samek and Sheremeta (2017):

All forms of recognition examined increase giving relative to the baseline treatment, and recognizing only the highest or only the lowest donors has the strongest and most significant effect.

20 of 21

Laboratory versus field experiments�

Laboratory experiment: an experiment conducted at a facility specifically designed for that purpose.

Field experiment: An experiment conducted within the setting of subjects’ everyday lives, possibly without their knowledge.

Lab and field experiments tend to have different advantages:

Which has greater internal validity?
Which is easier to replicate?
Which has greater external validity?

21 of 21

References�

Fehr, E., & Gachter, S. (2000). Cooperation and punishment in public goods experiments. American Economic Review, 90, 980-994.
Hertwig, R., & Ortmann, A. (2001) Experimental practices in economics: A methodological challenge for psychologists? Behavioral and Brain Sciences, 24, 383-451.
List, J.A. (2011). Why economists should conduct field experiments and 14 tips for pulling one off. Journal of Economic Perspectives, 25, 3-16.
Samek, A., & Sheremeta, R. M. (2017). Selective recognition: How to recognize donors to increase charitable giving. Economic Inquiry, 55, 1489-1496.
Smith, V.L. (1994). Economics in the laboratory. Journal of Economic Perspectives, 8, 113-131.