LO 4.2.2.I

Learning Objective: Describe the variable selection property of the Lasso.

Review:

• With reference to the figure shown below:

The ellipses that are centered around represent regions of constant RSS.

As the ellipses expand away from the least-squares coefficient estimates, the RSS increases.

Lasso and ridge regression coefficient estimates are given by the first point at which an ellipse contacts the green constraint region.

• Since ridge regression has a circular constraint region with no sharp points, this intersection will not generally occur on an axis, and so the ridge regression coefficient estimates will be exclusively non-zero.

• Conversely, the Lasso constraint region has corners at each of the axes, and so the ellipse will often intersect the constraint region at an axis. When this occurs, one of the coefficients will equal zero. Generalizing the approach for a  p-dimensional Lasso regression, then q of the coefficient estimates would equal zero simultaneously, while p-q coefficient estimates will be different from zero.

Source: Assigned reading