LO 4.2.2.F

Learning Objective: Describe how ridge regression improves upon least squares.

Review:

• Ridge regression works best in situations where the least-squares regression estimates have high variance.

• This occurs when a small change in the training data can cause a large change in the least-squares coefficient estimates.
• In particular, when the number of variables p is almost as large as the number of observations n, (p ≈ n), the least-squares regression estimates will be extremely variable.
• And if p > n, the least-squares regression estimates do not even have a unique solution, whereas ridge regression can still perform well by trading off a small increase in bias for a large decrease in variance.