Lecture 31

Least Squares

DATA 8

Fall 2023

Announcements

• Project 2 checkpoint due today at 11pm!
• Full project due Friday 11/10
• Keep in mind this is a holiday; try to submit by early deadline!
• Early submission bonus Thursday 11/9
• Homework 10 due 11/8
• Chapter 15 of the textbook is helpful!

​

Regression Roadmap

• Monday
• How to measure linear association
• Wednesday
• Predicting one numerical variable from a another
• The regression line
• Today
• The “best” linear predictor
• The method of least squares

Linear Regression

Prediction

Goal: Predict y using x

So far we have been using numerical variables but it is possible to use categorical variables in the regression as well

Examples:

• Predict # hospital beds available using air pollution

​

• Predict house prices using house size

​

• Predict # app users using # app downloads

​

Regression Estimate

Goal: Predict y using x

​

To find the regression estimate of y:

• Convert the given x to standard units
• Multiply by r
• That’s the regression estimate of y, but:
• It’s in standard units
• So convert it back to the original units of y

Slope and Intercept (in original units)

estimate of y = slope * x + intercept

Discussion Question

A course has a midterm (average 70; standard deviation 10)�and a really hard final (average 50; standard deviation 12)

If the scatter diagram comparing midterm & final scores for students has an oval shape with correlation 0.75, then...

What do you expect the average final score would be for students who scored 60 on the midterm?

Discussion Question

Suppose we use linear regression to predict candy prices (in dollars) from sugar content (in grams). What are the units of each of the following?

• r

​

• The slope

​

• The intercept

Least Squares

(Demo)

Error in Estimation

• error = actual value − estimate

​

• Typically, some errors are positive and some negative

​

• To measure the rough size of the errors
• square the errors to eliminate cancellation
• take the mean of the squared errors
• take the square root to fix the units
• root mean square error (rmse)

(Demo)

Numerical Optimization

• Numerical minimization is approximate but effective
• Lots of machine learning uses numerical minimization
• If the function mse(a, b)returns the mse of estimation using the line “estimate = ax + b”,
• then minimize(mse)returns array [a₀, b₀]
• a₀ is the slope and b₀ the intercept of the line that minimizes the mse among lines with arbitrary slope a and arbitrary intercept b (that is, among all lines)

(Demo)

Least Squares Line

• Minimizes the root mean squared error (rmse) among all lines

​

• Equivalently, minimizes the mean squared error (mse) among all lines

​

• Names:
• “Best fit” line
• Least squares line
• Regression line

Non-Linear Regression

(Demo)