Section 8a��Simple Linear Regression & Correlation��one continuous predictor (X),� one continuous outcome (Y)�

Ex: Riddle, J. of Perinatology (2006) 26, 556–561

50^th percentile for birth weight (BW) in g

as a function of gestational age

​

Birth Wt (g) =42 exp(0.1155 gest age)

or

Log_e(BW) = 3.74 + 0.1155 gest age

​

In general: BW = A exp(B gest age), A & B change for different percentiles

log(BW) = C + B gest age where C=log(A)

​

Example: Nishio et. al. Cardiovascular Revascularization Medicine 7 (2006) 54– 60

Simple Linear Regression statistics

Statistics for the association between a continuous X and a continuous Y.

A linear relation is given by an equation

Y = a + b X + errors (errors=e=Y-Ŷ)

Ŷ = predicted Y = a + b X

a = intercept, b =slope= rate of change

r = correlation coefficient, R²=r²

R²= proportion of Y’s variation due to X

SD_e=residual SD=RMSE=√mean square error

Ex: X=age (yrs) vs Y=SBP (mmHg)

SBP = 81.5 + 1.22 age + error

SD_e = 18.6 mm Hg, r = 0.718, R² = 0.515

“Residual” error

residual error = e = Y – Ŷ

​

The “residual error” is the difference between the actual observed value Y and the model (equation) predicted value, Ŷ.

The sum and mean of the e_i’s will always be zero. The residual error standard deviation, SD_e, is a measure of how close the observed Y values are to their equation predicted values (Ŷ). When r=R²=1, SD_e=0.

​

age vs SBP in women - Predicted SBP (mmHg) = 81.5 + 1.22 age, r=0.72, R²=0.515

Mean error is always zero

Confidence intervals (CI)�Prediction intervals (PI)

Model: predicted SBP=Ŷ=81.5 + 1.22 age

For age=50, Ŷ=81.5+1.22(50) = 142.6 mm Hg

95% CI: Ŷ ± 2 SEM, 95% PI: Ŷ ± 2 SD_e

SEM=3.3 mm Hg ↔ 95% CI is (136.0, 149.2)

SD_e=18.6 mm Hg ↔ 95% PI (104.8,180.4)

​

The Ŷ=142.6 is predicted mean for all age 50 and the predicted value for one individual age 50. Both interpretations are correct.

​

Model fit - R square statistic

Y = Ŷ + e

The variation in Y has two sources:

Variation of Ŷ – this is accounted for by X

Variation of e – this is NOT accounted for by X

total Y variation = Ŷ variation + e variation

R² = R square = Ŷ variation / total variation =

100 R² is the percent of the total variation

“accounted for” by X (or by Xs in multiple regr).

For SBP vs age, R²=0.515=51.5%

R squared statistic�measure of model accuracy

SD_y² = “all” the variation in Y

​

SD_e² = the variation NOT accounted for by X (more generally, not accounted for by Xs)

​

SD_y² – SD_e² = the variation that IS accounted for by X (or all the Xs)

​

R²=(SD_y² – SD_e²)/ SD_y² = proportion of total variation “accounted for” by Xs.

SD_e² = (1-R²) SD_y²

​

​

R² definition & interpretation

R² is the proportion of the total (squared) variation in Y that is “accounted for” by X.

R²= r² = (SD_y²– SD_e²)/SD_y² =1- (SD_e²/SD_y²)

SD_y√(1-r²) = SD_e

Under Gaussian theory, 95% of the errors are within +/- 2 SD_e of their corresponding predicted Y value, Ŷ.

​

Y=Ŷ+e, Var(Y) = Var(Ŷ+e) = Var(Ŷ) + Var(e)

Var(Ŷ) = Var(Y) – Var(e) = SD_y^{2 –}SD_e²

R²=Var(Ŷ)/Var(Y) = (SD_y^{2 –}SD_e²) / SD_y²

​

How big should R² be?

SBP SD = 26.4 mm Hg, SD_e=18.6

95% PI: Ŷ± 2(18.6) or Ŷ± 37.2 mm Hg

How big does R² have to be to make

95% PI: Ŷ ± 10 mm Hg? 🡪 SD_e≈ 5 mm Hg

​

R²=1-(SD_e/SD_y)²=

1-(5/26.4)² = 1-0.036=0.964 or 96.4%

(with age only, R² = 0.515)

Correlation coefficient (r)

Instead of regressing Y on X,

create Y_s = (Y-mean Y)/SD_y =“standardized Y”

X_s = (X-mean X)/SD_x =“standardized X”

Y_s & X_s are both in SD units & have mean=0

If regress Y_s on X_s: Y_s = a + b X_s + error

a is always 0. “b” is called “r”, the correlation coefficient, the “slope” in SD units.

r ranges from -1 to 1. r stays the same if the roles of Y_s and X_s are reversed. A value of r=0 implies no (linear) association between X and Y.

Correlation-interpretation, |r| < 1

Slope is related to correlation�(simple regression)

slope = correlation x (SD_y/SD_x)

​

b = r (SD_y/SD_x) b=1.22=0.7178(26.4/15.5)

​

where SD_y is the SD of the Y variable

SD_x is the SD of the X variable

​

r = b (SD_x/SD_y) 0.7178=1.22(15.5/26.4)

​

r = b SD_x/√ b² SD_x² + SD_e²

​

SD_e = residual error SD, SD_x = SD of X

when b=0, r=0, when b>0, r>0, when b<0, r<0

Computing slope & correlation

Pearson vs Spearman corr=r

Pearson r – Assumes relationship between Y and X is linear except for noise.

“parametric” (inspired by bivariate normal model). Strongly affected by outliers.

Spearman r_s – Based on ranks of Y and X. Assume relation between Y and X is monotone (non increasing, non decreasing). “Non parametric”. Less affected by outliers.

Pearson vs Spearman (rank)

Pearson r vs Spearman r_s

n=9, r =0.25, r_s = 0.48

Limitation of linear models

Mean increase in X=19.9

Mean increase in Y= 39.9

Do X and Y have a positive correlation?

​

Example: X is the increase in minutes of exercise. Y is the increase in Math test scores in the same group.

Mean changes in X and Y do not necessarily imply correlation

Mean changes do not imply correlation�

Just because there is a positive mean X change and a positive mean Y change does not necessarily imply that X and Y are correlated.

mean X change =19.9, mean Y change=39.9

r=0

Limitations of Linear Statistics�Example of a nonlinear relationship

Correlations are misleading if relation is not linear or at least monotone

Limits to linear regression – “Pathological” Behavior��Ŷ = 3 + 0.5 X, r = 0.817, SDe = 13.75, n=11�(for all four datasets below)�

​

​

​

Weisberg, Applied Linear Regression, p 108

Simpson’s paradox

Final grade (Y) vs hours studying (X), r = -0.7981 – negative relationship ?

Simpson’s paradox (cont.)

Controlling for type of course – relations are now positive

“Ecologic” fallacy

Wrong unit of analysis-City, not person

Ecologic Fallacy- Must look at the correct unit of analysis

truncating X, true r=0.9, R²=0.81

Full data

Interpreting correlation in experiments

Since r=b(SD_x/SD_y), an artificially lowered SD_x will also lower r.

​

R², b and SD_e when X is systematically changed

​

Data R² b SD_e

Complete data 0.81 0.90 0.43

(“truth”)

​

Truncated 0.47 1.03 0.43

(X < -1 SD deleted)

​

center deleted 0.91 0.90 0.45

( -1 SD< X < 1 SD deleted)

​

extremes deleted 0.58 0.92 0.42

(X < -1 SD deleted, X > 1 SD deleted)

​

Assumes intrinsic relation between X and Y is linear except for error.

Attenuation of regression coefficients�when there is random error in X (true slope=β= 4.0)

Negligible errors in X:

Y=1.149 + 3.959 X

SE(b) = 0.038

Noisy errors in X:

Y=-2.132 + 3.487 X

SE(b) = 0.276

Attenuation (random error in X) drives the statistic toward its null value.

Example:

X=smoking (y/n), Y=lung cancer (y/n)

Compute odds ratio = OR

Randomly misclassify smokers/non smokers

What happens to the OR?

OR gets closer to 1.0 (conservative)

​

Attenuation gives conservative results.

​

Checking for linearity – smoothing & splines

Basic idea: In a plot of Y vs X, also plot Ŷ vs X where

Ŷ_i = ∑ W_ni Y_i where ∑ W_ni=1, W_ni>0.

The “weights” W_ni, are larger near Y_i and smaller far from Y_i.

​

Smooth: define a moving “window” of a given width around the i^th data point and fit a mean (weighted moving average) in this window.

​

Spline: break the X axis into non-overlapping bins and fit a polynomial within each bin such that the “ends” all “match”.

​

The size of the window or bins control the amount of smoothing.

We smooth until we obtain a smooth curve but go no further.

Check for linearity - Smoothing

1. Make a bin (window), take a (possibly weighted) average of all Y values in the bin or carry out a regression in the bin. The predicted value (mean) in the middle of the bin is the smoothed value (Ŷ).

2. Move bin over (overlapping) and repeat.

3. Connect the predicted values (Ŷs) with lines across all bins.

​

34

Smoothing to check for linearity�(kernel smoothing)

35

Move window over- compute smoothed value

36

Move window over- compute smoothed value

37

Splines to check for linearity

Spline- Break the X axis into equally spaced non overlapping “bins” (segments). Fit a polynomial (usually a quadratic or cubic) within each bin such that Y at the “ends” (“knots”) all “match” (are piecewise continuous) and their first derivatives (slopes) are also continuous.

38

Draftsman’s spline

39

Splines vs smoothing �(JMP-Fitting linear models)

1. Difficult to carry out smoothing in high dimensions (more than one X). That is, smoothing is essentially bivariate, not multivariate.
2. Smoothing is completely defined by Y. Spline functional form is independent of Y.
3. Splines only add k-2 parameters for k knot points.
4. Splines are the “natural” test for curvature vs linearity.

​

40

Spline example-constant spline�same as “polychotomizing” x

41

Linear and (restricted) cubic spline

42

Generally use restricted cubic splines with 3-5 knots to check linearity

Restricted cubic spline-RCS

Cubic equation: Ŷ = b₀ + b₁X + b₂X² + b₃X³

Fit an equation within each non overlapping bin (segment).

​

Restrictions on b₀, b₁, b₂, b₃:

Ŷ must be same in adjacent bins at the (X) knots.

dŶ/dX = b₁ + 2b₂X + 3b₃X²

must be the same in adjacent bins at the knots (makes “smooth” connection).

Checking for linearity via spline

Fit a straight line to the data (linear model)

Fit a restricted cubic spline to the same data (non linear model). Compare fit (R square, SD_e). F test for this comparison:

F=([RSS_linear-RSS_spline]/k)/(SDe_spline)²

If the straight line fits about as well as the cubic spline, (p value from F not significant) can say the relationship between Y and X is linear. (k=df spline-df linear, RSS= residual error sum of squares)

RSS_e = (n-k) SD_e²

Smoothing example�IGFBP by BMI

Smoothing

Insufficient smoothing

Over smoothing

IGFBP by BMI

Smoothing example�IGFBP by BMI

Smoothing

Smoothing example�IGFBP by BMI

Insufficient smoothing

Smoothing example�IGFBP by BMI

Over smoothing

ANDRO by BMI – is underlying relationship linear?

Check linearity - ANDRO by BMI�assume linear, R²=0.153�assume spline, R²=0.165�p value = 0.8130 (LR test, null is linear)

Linear

Spline