1

Applied Data Analysis (CS401)

Maria Brbic / Robert West

Lecture 6

Causal analysis of observational data

16 Oct 2024

Announcements

2

• Homework H1 due on Fri Oct 18^th 23:59
• You can ask questions until Thu 23:59; we won’t answer questions asked after Thu
• Project:
• Milestone P1 feedback has been released
• Milestone P2 get cracking once homework H1 is done!
• Don’t use ChatGPT
• Friday’s lab session:
• Exercises on causal analysis of observational data (Exercise 5)

Course evaluation

• Thanks for your feedback! -- We’ll use it to improve the class further
• Most of you like the class and what you learn. Yaaay! :)
• Some concerns: information not centralized (moodle, website, ed, google forms etc), lectures cover many topics but not going deep, more help during the exercise session

3

Feedback

4

Give us feedback on this lecture here: https://go.epfl.ch/ada2024-lec6-feedback

​

• What did you (not) like about this lecture?
• What was (not) well explained?
• On what would you like more (fewer) details?
• …

​

Dr. Maria’s smoking cure

• I claim to have developed a medicine that helps you quit smoking
• I ask all smokers: “Do you want to try my medicine?”
• Smokers = {treated smokers}� ⋃ {untreated (“control”) smokers}
• Fraction of successful quitters is higher in the treated group
• I conclude: “My medicine helps you quit smoking! Buy it!”
• Do you believe me?

5

Goals of this lecture

• Clarify difference between experimental and observational studies
• Highlight pitfalls of observational studies
• Give you tools for avoiding the pitfalls, allowing you to draw valid conclusions from “found data” (very useful for project!)
• Motivate you to read Rosenbaum’s great book “Design of Observational Studies” (in particular Chapters 1, 2, and 3; or this book) and Pearl’s eye-opening “Book of Why”

6

Dr. Maria’s “experiment” as a causal diagram

7

Motivation to quit smoking etc.

Take Dr. Maria’s medicine

Quit smoking

“Confounder”💀💀💀

“Treatment”

“Outcome”

Dr. Maria’s “experiment” as a causal diagram

8

Motivation to quit smoking etc.

Quit smoking

“Confounder”💀💀💀

“Treatment”

“Outcome”

Take Dr. Maria’s medicine

Ideal setting as a causal diagram

9

Motivation to quit smoking etc.

Take Dr. Maria’s medicine

Quit smoking

Randomized controlled experiments

• Two experimental conditions:
• Treatment (e.g., medicine)
• Control (e.g., placebo [fun fact])
• Assignment of participants to conditions is random
• Probability of receiving treatment same for everyone
• Treatment and control groups are indistinguishable
• E.g., determination to quit smoking is not systematically higher in the treated group

10

Randomized controlled experiments

11

 by Jonathan Pierrel

Randomized controlled experiment as a causal diagram

12

12

Motivation to quit smoking etc.

Take Dr. Maria’s medicine

Quit smoking

Randomized controlled experiment as a causal diagram

13

13

Motivation to quit smoking etc.

Take Dr. Maria’s medicine

Quit smoking

Randomized controlled experiment as a causal diagram

14

14

Motivation to quit smoking etc.

Take Dr. Maria’s medicine

Quit smoking

Coin

Limits of randomization

• Do seat belts save lives?
• Experiment:
• Flip coin at birth to assign to treatment (always wear seat belt for entire life) or control (never wear seat belt)
• Measure fraction of traffic deaths in each group
• Randomized experiments aren’t always feasible
• Unethical (see above), expensive, fundamentally impossible (e.g., do earthquakes decrease life spans?)
• Most modern “big data” is “found data”
• Experiments may lead to unrealistic scenarios

15

Alternative: observational studies

• Fundamentally different from experiment:
• Researcher can’t control who goes to which condition
• Researcher is merely an observer, not a tinkerer
• Much less problematic w.r.t. ethics, price, feasibility
• Much more problematic w.r.t. validity of conclusions
• All advantages of randomized experiment are gone
• Subjects self-select to be treated
• Treatment assignment and response may be caused by same hidden correlate (a.k.a. confounders; e.g., motivation to quit smoking)

16

Example: seat belts revisited

• Recall: experiment infeasible because unethical
• Observational study:
• Dataset: all traffic accidents in a given time span
• Two treatment conditions:
• Treated: seat-belt wearers
• Control: non-seat-belt wearers
• Compare fraction dead in treated vs. control
• What problems do you see?

17

18

THINK FOR A MINUTE!

​

(Feel free to discuss with your neighbor.)

As a causal diagram

19

Driver’s cautiousness

Seat belt

Survival

Severity of crash

?

A matched observational study

• Consider only a particular subset of accident cars:
• 2 people in car: driver + passenger
• Exactly one of them died in accident
• Exactly one of them wore seat belt at time of accident (i.e., 1 treated + 1 control per car)
• As before: compare fraction dead in treated vs. control
• New: many potential confounders are controlled for,�incl. type of car, speed, severity of accident
• Fundamental concept: matching

20

As a causal diagram

21

21

Driver’s cautiousness

Seat belt

Survival

Severity of crash

Settling the seat-belt question

22

Natural experiments

• Not researcher, but nature, “flips a coin” to decide treatment assignment
• Rosenbaum: “When investigators are especially proud, having found unusual circumstances in which treatment assignment, though not random, seems unusually haphazard, they may speak of a ‘natural experiment.’”
• Examples: twin studies, Vietnam draft, cholera in London
• Is matched seat-belt study a natural experiment?

23

24

Nature didn’t flip a coin for me – should I just go home and weep?

25

Commercial break

26

Don’t go home and weep!

Nature didn’t flip a coin for me – should I go home and weep?

• No! You can still get good mileage if you’re smart about it
• Fundamental concept: matching
• Ideally: Pair up 2 identical people:
• 1 treated, 1 control
• Ex-post (vs. natural experiment: ex-ante)
• Compare outcome of treated vs. control
• e.g., mean difference treated-minus-control
• or regression analysis (see last lecture)

27

Matching

• Ideally: Pair up 2 identical people:
• 1 treated, 1 control
• Such ideal matching usually not feasible
• Problem 1: Unobserved covariates:�You usually can’t even know if two people are identical
• Problem 2: Combinatorial explosion:�(Nearly) no two people are identical

28

Problem 1: Unobserved covariates

• You usually can’t even know if two people are identical
• e.g., (hypothetical) gene that causes both desire to smoke�and lung cancer

29

Addressing Problem 1 by ignoring it:�A naive model

30

“People who look comparable are comparable”

“Only observed covariates determine treatment assignment”

or equivalently:

Naive model as a causal diagram

31

Observed covariates

Treatment

Outcome

Unobserved covariates

If the naive model was true…

• … you could “simulate” a randomized experiment:
• Simply match subjects with identical observed covariates (1 treated, 1 control)
• Subjects in a pair have the same probability to treat
• So who gets treated is up to chance, as in experiment
• Analysis: compare outcome for treated to outcome of control (e.g., mean difference treated-minus-control)

32

Problem 2!

Problem 2: Combinatorial explosion

• (Nearly) no two people are identical
• So finding two people to match is often impossible
• Even when considering only observed covariates�(as in the naive model)
• Do we really need to match people with identical covariates?
• Recall “holy grail”: randomized controlled trials
• Coin makes sure everyone has �identical probability to be treated
• → Let’s mimic what the coin does!

33

Addressing Problem 2: Propensity score

• Compress the (potentially many) observed covariates into a single number: the probability to receive the treatment (a.k.a. propensity score):� Pr(subject is treated | observed covariates)

​

• Can be estimated from the data
• E.g., via logistic regression (see next lecture)
• Input: observed covariates
• Output: treatment indicator (1 if treated, 0 if control)

34

Propensity score as a causal diagram

35

Propensity score

Treatment

Outcome

Observed covariates

…

Balancing property of propensity score

• Fact: all subjects (treated and control) with equal propensity score (PS) p have equal distribution of observed covariates x:�� Pr(x | treated = 1, PS = p) = Pr(x | treated = 0, PS = p)�
• Subjects in a matched pair might not have equal x, but treated and control groups will have similar distributions of x

36

Balancing property is propensity score’s reason to be

• There are many other methods for achieving balance
• e.g., exact matching, Mahalanobis distance matching, coarsened exact matching, …
• You can mix and match methods (e.g., match exactly on gender, use propensity scores for other covariates)
• What eventually matters is whether you achieve balance
• Regardless of how you try to achieve balance, you need to verify that you managed to achieve balance (p.t.o.)

37

Assessing covariate balance

38

Before matching

After matching

Matching algorithms

• Goal: Match subjects into pairs (1 treated, 1 control), with identical propensity scores within each pair
• Unlikely that 2 subjects have identical propensity scores
• → Use approximate matching (remember your algo class!)
• Bipartite graph: each subject connected to all other subjects
• Edge weights: absolute (or squared) difference of propensity scores (or other matching criterion)
• Find minimum matching,�e.g., via Hungarian algorithm

39

Ok, so are we done?

40

If the naive model isn’t true...

• … propensity score may differ from true probability to treat:��Pr(treated | observed covariates) ≠ Pr(treated | all covariates)

​

41

have this

need this

Violated naive model as a causal diagram

42

Treatment

Outcome

Observed covariates

…

Propensity score

If the naive model isn’t true...

… you may end up matching

King

The naive model is easily attacked

Rosenbaum:

44

The sensitivity analysis model

• Idea: Quantify the degree to which the naive model may be wrong without you having to change your (causal) conclusions
• Assume that treatment odds of identical-looking subjects (i.e., identical observed covariates x) may differ by up to a factor Γ
• Then reason in spirit of proof by contradiction: “To change the conclusions of my study, two identical-looking people (1 treated, 1 control) would have to have hugely different treatment odds (i.e., huge Γ). Common sense (or domain knowledge) suggests that this is not the case, so my conclusions stand.”

45

The sensitivity analysis model

• Bounded odds ratio (OR)
• Reason for using OR:�OR = Pr(k treated | either k or ℓ treated) / Pr(ℓ treated | either k or ℓ treated)
• Sensitivity Γ = 1 → naive model is true
• Sensitivity Γ = 2 → subject with same observed covariates x up to twice as likely to be the one to receive treatment
• Sensitivity Γ = ∞ → void statement (a.k.a. tautology)

46

subject ℓ’s (true) probability to treat

Example: smoking and lung cancer

• Under naive model: matching on observed covariates gives a very small p-value for the null hypothesis that smoking does not increase lung cancer risk (using an appropriate hypothesis test), i.e., data hard to explain w/o a causal effect
• Tobacco lobby: “The naive model isn’t true! There may be hidden (e.g., genetic) correlates that increase both the probability to enjoy smoking and the probability of lung cancer. They, not smoking, cause cancer!”

47

Example: smoking and lung cancer

• Under sensitivity analysis model, increasing sensitivity Γ increases the p-value for null hypothesis
• Anti-tobacco lobby: “But making p > 0.05 would require�Γ > 6; i.e., the odds of being a smoker would need to be six times higher for one of two people with the exact same observed features (age, gender, education, income, ...). It’s unlikely that any unobserved covariate would have such a large effect on smoking habits. So smoking causes cancer!”

48

Two parts: mechanical vs. scientific

• Mechanical part:
• Create pairs (1 treated + 1 control) with similar observed covariates (using exact or propensity-score matching)
• Scientific (i.e., fun) part:
• Mitigate concerns that your findings might be caused by unobserved covariates, rather than treatment (e.g., using sensitivity analysis, ad-hoc arguments, natural experiments)

49

Summary

• Holy grail: randomized experiment
• When experiment not possible: observational study
• Crucial problem: treatment assignment not random (biases!)
• Semi-holy grail: natural experiment
• Matched studies: pair up treated/control based on observed covariates
• Problem: still, treatment assignment not random (biases via unobserved covariates)
• Solution: sensitivity analysis
• Keep this lecture (more here) in mind for your projects!

50

Feedback

51

Give us feedback on this lecture here: https://go.epfl.ch/ada2024-lec6-feedback

​

• What did you (not) like about this lecture?
• What was (not) well explained?
• On what would you like more (fewer) details?
• …

​

Credits

• Much of the material is based on Paul Rosenbaum’s amazing book “Design of Observational Studies”, available for free here

52