1

Applied Data Analysis (CS401)

Maria Brbic / Robert West

Lecture 5

Regression for disentangling data

09 Oct 2024

Linear regression as you know it

• Given: n data points (X_i, y_i), where X_i is k-dimensional vector of predictors (a.k.a. features) of i-th data point, and y_i is scalar outcome
• Goal: find the optimal coefficient vector 𝛽 = (𝛽₁, …, 𝛽_k) for approximating the y_i’s as a linear function of the X_i’s:���where 𝜖_i are error terms that should be as small as possible
• X_i1 usually the constant 1 (by def) ⇒ 𝛽₁ a constant intercept

2

Example with one predictor

3

X

y

𝛽₁: intercept

𝛽₂: slope

y ≈ 𝛽₁ + 𝛽₂X

3

Linear regression as you know it

• Given: n data points (X_i, y_i), where X_i is k-dimensional vector of predictors (a.k.a. features), and y_i is scalar outcome, of i-th data point
• Goal: find the optimal coefficient vector 𝛽 = (𝛽₁, …, 𝛽_k) for approximating the y’s as a linear function of the X’s:���where 𝜖_i are error terms that should be as small as possible
• X_i1 usually the constant 1 → 𝛽₁ a constant intercept

4

Optimality criterion: least squares

• Intuitively, want errors 𝜖_i to be as small as possible
• Technically, want sum of squared errors as small as possible�⇔ find such that we minimize

5

Use cases of regression

• Prediction: use fitted model to estimate outcome y for a new X not seen during model fitting (if you’ve seen regression before, then probably in the context of prediction)
• Descriptive data analysis: compare average outcomes across subgroups of data (today!)
• Causal modeling: understand how outcome y changes when you manipulate predictors X (next lecture is about causality, although not primarily using regression)

6

Regression as comparison of�average outcomes

7

Example with one binary predictor X_i

• X_i = mom_hs = “Did mother finish high school?” ∈ {0, 1}
• y_i = kid_score = child’s score on cognitive test ∈ [0, 140]

y_i = 𝛽₁ + 𝛽₂X_i + 𝜖_{i .}

kid_score = 78 + 12 · mom_hs + error

8

0

1

mom_hs

kid_score

20 60 100 140

No

Yes

mean kid_score for moms who didn’t finish high school: 78

mean kid_score for moms who finished high school: 78 + 12 = 90

One binary predictor X_i:�Interpretation of fitted parameters 𝛽

y_i = 𝛽₁ + 𝛽₂X_i + 𝜖_{i .}

• Intercept 𝛽₁: mean outcome for data points i with X_i = 0
• Slope 𝛽₂: difference in mean outcomes between data points with X_i = 1 and data points with X_i = 0
• Reason: means minimize least-squares criterion:�∑ⁿ_i=1 (y_i – m)² is minimized w.r.t. m when�–2 ∑ⁿ_i=1 (y_i – m) = 0, i.e., when m = (1/n) ∑ⁿ_i=1 y_i

9

One binary predictor X_i:�Interpretation of fitted parameters 𝛽

y_i = 𝛽₁ + 𝛽₂X_i + 𝜖_{i .}

• Intercept 𝛽₁: mean outcome for data points i with X_i = 0
• Slope 𝛽₂: difference in mean outcomes between data points with X_i = 1 and data points with X_i = 0
• Reason: means minimize least-squares criterion:�∑ⁿ_i=1 (y_i – m)² is minimized w.r.t. m when�–2 ∑ⁿ_i=1 (y_i – m) = 0, i.e., when m = (1/n) ∑ⁿ_i=1 y_i

10

So why not just compute the two means separately and then compare them?

What a mean monkey!

Example with one continuous predictor X_i

• X_i = mom_iq = mother’s IQ score ∈ [70, 140]
• y_i = kid_score = child’s score on cognitive test ∈ [0, 140]

y_i = 𝛽₁ + 𝛽₂X_i + 𝜖_{i .}

kid_score = 26 + 0.6 · mom_iq + error

11

mom_iq

kid_score

0 50 100

estimated (hypothetical) mean kid_score for moms with IQ = 0: 26

estimated mean kid_score for moms with IQ = 100: 26 + 0.6 · 100 = 86

0 50 100 150

One continuous predictor X_i:�Interpretation of fitted parameters 𝛽

y_i = 𝛽₁ + 𝛽₂X_i + 𝜖_{i .}

• Intercept 𝛽₁: estimated mean outcome for data points i�with X_i = 0
• Slope 𝛽₂: difference in estimated mean outcomes between data points whose X_i’s differ by 1
• Why “estimated”? → e.g.,�
• NB: for binary predictor, we got “exact” instead of “estimated”

12

Example with multiple predictors

• (X_i1 = 1 = constant)
• X_i2 = mom_hs = “Did mother finish high school?” ∈ {0, 1}
• X_i3 = mom_iq = mother’s IQ score ∈ [70, 140]
• y_i = kid_score = child’s score on cognitive test ∈ [0, 140]

y_i = 𝛽₁ + 𝛽₂X_i2 + 𝛽₃X_i3 + 𝜖_{i .}

kid_score = 26 + 6 · mom_hs + 0.6 · mom_iq + error

No

Yes

13

Example with multiple predictors

kid_score = 26 + 6 · mom_hs + 0.6 · mom_iq + error

mom_iq

kid_score

20 60 100 140

80 100 120 140

kids of moms who didn’t finish high school:

intercept = 26

slope = 0.6

kids of moms who finished high school:

intercept = 26 + 6 = 32

slope = 0.6

14

Example with interaction of predictors

• X_i2 = mom_hs = “Did mother finish high school?” ∈ {0, 1}
• X_i3 = mom_iq = mother’s IQ score ∈ [70, 140]
• y_i = kid_score = child’s score on cognitive test ∈ [0, 140]

y_i = 𝛽₁ + 𝛽₂X_i2 + 𝛽₃X_i3 + 𝛽₄X_i2X_i3 + 𝜖_{i .}

kid_score = −11 + 51 · mom_hs + 1.1 · mom_iq − 0.5 · mom_hs · mom_iq + error

No

Yes

15

Example with interaction of predictors

kid_score = −11 + 51 · mom_hs + 1.1 · mom_iq − 0.5 · mom_hs · mom_iq + error

mom_iq

kid_score

20 60 100 140

80 100 120 140

kids of moms who didn’t finish high school:

intercept = −11

slope = 1.1

kids of moms who finished high school:

intercept = −11 + 51 = 40

slope = 1.1 − 0.5 = 0.6

16

17

So why not just compute the two means separately and then compare them?

Mom finished high school

Mom�didn’t finish high school

Mom drives Mercedes

Mom doesn’t drive Mercedes

Mom finished high school

Mom�didn’t finish high school

Mom drives Mercedes

Mom doesn’t drive Mercedes

18

19

THINK FOR A MINUTE:

​

What is the mean outcome for Mercedes-driving moms vs. for non-Mercedes-driving moms?�Compare the two means! What does the comparison tell you about the link between Mercedes-driving and kid_score?

​

(Feel free to discuss with your neighbor.)

• Mean kid_score for Mercedes drivers: 0.99 · 90 + 0.01 · 78 ≈ 90
• Mean kid_score for non-Mercedes drivers: 0.01 · 90 + 0.99 · 78 ≈ 78
• But really driving Mercedes makes no difference (for fixed high-school predictor)!
• Root of evil: correlation between finishing high school and driving Mercedes
• Regression to the rescue: kid_score = 78 + 12 · mom_hs + 0 · mercedes + error

Mom finished high school

Mom�didn’t finish high school

Mercedes

Mom finished high school

Mom�didn’t finish high school

No Mercedes

Mercedes

No Mercedes

20

Course eval (“indicative feedback”) open until �Sun 13 Oct 13^th �Go to https://isa.epfl.ch now!

21

Instructions: https://www.epfl.ch/education/teaching/teaching-support/resources-for-students/#indicativefeedback

Quantifying uncertainty

22

Quantifying uncertainty

• Statistical software gives you more than just coefficients 𝛽:

23

Residuals and R²

• Residual for data point i: estimation error on data point i:

​

• Mean of residuals = 0�(total overestimation = total underestimation)
• Variance of residuals�= avg squared distance of predicted value from observed value�= “unexplained variance”
• Fraction of variance explained by the model:

Variance of�outcomes y

24

Residuals and R²

• Residual for data point i: estimation error on data point i:

​

• Mean of residuals = 0�(total overestimation = total underestimation)
• Variance of residuals�= avg squared distance of predicted value from observed value�= “unexplained variance”
• Fraction of variance explained by the model:

Variance of�outcomes y

25

Coefficient of determination: R²

R² = 0.147

R² = 0.865

26

Coefficient of determination: R²

27

Coefficient of determination: R²

R² = 0.67 everywhere!

28

Assumptions made in regression modeling

29

Assumptions for regression modeling

1. Validity:
1. Outcome measure should accurately reflect the phenomenon of interest
2. Model should include all relevant predictors
3. Model should generalize to cases to which it will be applied

30

Assumptions for regression modeling (2)

2. Linearity:

��But very flexible: we require linearity in predictors (not necessarily in raw inputs); predictors can be arbitrary functions of raw inputs, e.g.,�- logarithms, polynomials, reciprocals, … �- interactions (i.e., products) of multiple inputs�- discretization of raw inputs, coded as indicator variables

31

Assumptions for regression modeling (3)

3. Independence of errors: no interaction between data points
4. Equal variance of errors
5. Normality (Gaussianity) of errors

less important�in practice

32

Transformations of predictors and outcomes

33

Transformations of predictors

• When we apply linear transformations to predictors, the model remains “equally good”:
• The fitted coefficients may change, but predicted outcomes and model fit (R²) won’t change
• For instance,

34

Mean-centering of predictors

• Compute the mean value of a predictor over all data points, and subtract it from each value of that predictor:�X_ij ← X_ij − mean(X_1j, …, X_nj)
• ⇒ the predictor X_ij now has mean 0

-100 -50 0 50

mean kid_score for moms with mean IQ: 86

0 50 100

mom_iq

kid_score

0 50 100

(hypothetical) mean kid_score for moms with IQ = 0: 26

0 50 100 150

35

After mean-centering of predictors, …

… you have a convenient interpretation of coefficients 𝛽_j of main predictors (i.e., non-interaction predictors):

• j = 1 (i.e., intercept):
• Estimated mean outcome when each predictor has its mean value
• j > 1:
• Model w/o interactions: estimated mean increase in outcome y for each unit increase in X_ij
• Model with interactions: estimated mean increase in outcome y for each unit increase in X_ij when each other predictor has its mean value

36

Standardization via z-scores

• First mean-center all predictors, then divide them by their standard deviations:�X_ij ← [X_ij − mean(X_1j, …, X_nj)] / sd(X_1j, …, X_nj)
• Resulting values are called “z-scores”
• All predictors now have the same units:�distance (in terms of standard deviations) from the mean
• This lets us compare coefficients for predictors with previously incomparable units of measurement, e.g., IQ score vs. earnings in Swiss francs vs. height in centimeters

37

Logarithmic outcomes

• Practical: makes sense if the outcome y�follows a heavy-tailed distribution
• Only works for non-negative outcomes
• Theoretical: turns an additive model�into a multiplicative model:

38

Logarithmic outcomes: Interpreting coefficients

• An additive increase of 1 in predictor X_·1 is associated with a multiplicative increase of B₁ := exp(b₁) in the outcome
• If b₁ ≈ 0, we can immediately interpret b₁ (without needing to exponentiate it first to get B₁!) as the relative increase in outcomes, since exp(b₁) ≈ 1 + b₁
• E.g., b₁ = 0.05 ⇒ B₁ = exp(b₁) ≈ 1.05�⇒ “+1 in predictor X_·1” is associated with “+5% in outcome”

39

Going beyond linear regression for comparing means

40

Beyond linear regression:�generalized linear models

• Logistic regression: binary outcomes
• Poisson regression: non-negative integer outcomes (e.g., counts)

41

Beyond comparing means; or, A taste of causality: “Difference in differences”

• Two groups: P, S
• At time 2, group P receives a treatment, group S doesn’t
• Question: Did the treatment have an effect? If so, how large was it?
• P and S don’t start out the same at time 1
• There is a temporal “baseline effect” even w/o treatment

42

Beyond comparing means; or, A taste of causality: “Difference in differences” (2)

• Elegant linear model with binary predictors:�y_it = a + b · treated_i + c · time2_t� + d · (treated_i · time2_t) + error_i
• d = treatment effect
• All of this with one single regression!
• You get quantification of uncertainty (significance) for free!�

43

a

b

c

d

Beyond comparing means; or, A taste of causality: “Difference in differences” (2)

• Elegant linear model with binary predictors:�y_it = a + b · treated_i + c · time2_t� + d · (treated_i · time2_t) + error
• d = treatment effect
• All of this with one single regression!
• You get quantification of uncertainty (significance) for free!�

44

a

b

c

d

A bonanza of causality:

Next lecture!

45

$#1t!, my banana is non-linear…

Summary

• Linear regression as a tool for comparing means across subgroups of data
• How? Read group means off from fitted coefficients
• Advantages over plain comparison of means “by hand”:
• Accounting for correlations among predictors
• Quantification of uncertainty (significance) “for free”
• Additive vs. multiplicative model: all it takes is a log
• Caveat emptor:
• Model must be appropriately specified, else nonsense results → stay critical, run diagnostics (e.g., R², data viz)

46

Feedback

47

Give us feedback on this lecture here: https://go.epfl.ch/ada2024-lec5-feedback

​

• What did you (not) like about this lecture?
• What was (not) well explained?
• On what would you like more (fewer) details?
• …

​

Credits

• Much of the material in this lecture is based on Andrew Gelman and Jennifer Hill’s great book “Data Analysis Using Regression and Multilevel/Hierarchical Models”, available for free here
• For a neat and gentle written intro to linear regression, especially check out chapters 3 and 4

48

Bonus: Logarithmic outcomes and predictors

Interpretation of coefficient of logarithmic predictor:

• Multiplicative increase by 1% in predictor X_·1 is associated with a multiplicative increase by b₁% in the outcome
• Why?
• log(y) = a + b log(X) ⇒ y = exp(a) * X^b
• Multiplying X by a factor c multiplies y by a factor of c^b
• c^b ≈ 1 + b*(c–1) for c ≈ 1 (hint: Taylor approximation!)
• Example when using c = 1.01 (i.e., increase by 1%):�b = 2 ⇒ increasing X by 1% increases y by 2%

49