Lecture 19

Regression Inference

Regression Model

A “Model”: Signal + Noise

Distance drawn at random from normal distribution with mean 0

Another distance drawn independently from the same normal distribution

What We Get to See

(Demo)

How do we assess performance?

i.e., Goodness of Fit

Variance of Fitted Values

• Variance = Square of the SD

= Mean Square of the Deviations

• Variance has weird units, but good math properties

​

• Variance of fitted values

--------------------------------- = r²

Variance of y

Coefficient of Determination

(Demo)

A Variance Decomposition

​

• Variance of fitted values

--------------------------------- = r²

Variance of y

​

• Variance of residuals

--------------------------------- = 1 - r²

Variance of y

Residual Average and SD

• The average of residuals is always 0

​

• Variance of residuals

--------------------------------- = 1 - r²

Variance of y

​

• SD of residuals = √(1 - r²) SD of y

The True Slope

(Demo)

Confidence Interval for True Slope

• Bootstrap the scatter plot.
• Find the slope of the regression line through the bootstrapped plot.
• Repeat.
• Draw the empirical histogram of all the generated slopes.
• Get the “middle 95%” interval.
• That’s an approximate 95% confidence interval for the slope of the true line.

(Demo)

Rain on the Regression Parade

We observed a slope based on our sample of points.

But what if the sample scatter plot got its slope just by chance?

What if the true line is actually FLAT?

(Demo)

Test Whether There Really is a Slope

• Null hypothesis: The slope of the true line is 0.
• Alternative hypothesis: No, it’s not.
• Method:
• Construct a bootstrap confidence interval for the true slope.
• If the interval doesn’t contain 0, reject the null hypothesis.
• If the interval does contain 0, there isn’t enough evidence to reject the null hypothesis.

Uncertainty in Prediction

Regression Prediction

• If the data come from the regression model,
• and if the sample is large, then:

​

• The regression line is close to the true line
• Given a new value of x, predict y by finding the point on the regression line at that x

(Demo)

Confidence Interval for Prediction

• Bootstrap the scatter plot
• Get a prediction for y using the regression line that goes through the resampled plot
• Repeat the two steps above many times
• Draw the empirical histogram of all the predictions.
• Get the “middle 95%” interval.
• That’s an approximate 95% confidence interval for the height of the true line at y.

Predictions at Different Values of x

• Since y is correlated with x, the predicted values of y depend on the value of x.

​

• The width of the prediction interval also depends on x.
• Typically, intervals are wider for values of x that are further away from the mean of x.

Summary

• We only get to see the data. We are working “as if” a linear model generated the data.
• r^2 tells us how much variability a linear model explains in the data.
• Goodness of fit.
• Because we only see a sample of data, we have uncertainty in
• The estimate of the slope.
• The predicted value.