Cross-Validation,�Regularization

(Reading: 15.3, Ch 16)

(Slides adapted from Sandrine Dudoit and Joey Gonzalez)

UC Berkeley Data 100 Summer 2019

Sam Lau

Learning goals:

• Learn how to perform K-fold CV and its benefits over a held-out validation set.
• Understand L2 and L1 regularization and how to use regularization to manage the bias-variance tradeoff.

Announcements

• HW5 out, due tomorrow
• HW6 out tomorrow, due Tuesday
• Screencast yesterday got frozen but audio is there
• If you leave a comment on the YT video with the slide numbers and times I can update the description, e.g.
• 00:00 - Slide 1�01:30 - Slide 2�etc.

Cross-Validation

Simple Validation

Sample

Used to fit a model.

Used to choose a model.

Used to report final accuracy.

Assessing Model Risk

Training Set

Validation Set

Test Set

Training Error

Validation Error

Test Error

Used to fit a model.

Used to choose a model.

Used to report final accuracy.

Minimizes empirical risk

Estimates population risk

“Clean” estimate of population risk

Model Selection

• E.g. f¹ is linear, f² is deg 2 poly, f³ is linear with fewer features, etc.
• Fit θ for each model by minimizing the training error.
• Compute validation error for each model.
• Pick the model with the lowest validation error.
• This is model selection.
• Now, report the test error of chosen model.

• Given models:

K-Fold CV

• Intuition: Validation error will not always be close to true risk. (Sometimes we are just unlucky!)
• To address, compute multiple validation errors for each model.
• K-Fold cross-validation:
• Set aside test set from sample.
• Split sample into K equal sized partitions
• Use K - 1 splits to train, last split as validation set.
• Repeat K times, average of K errors is validation error.

3-Fold CV

Sample

We repeat this entire process for each model we want to try out.

K-Fold CV Analysis

• K usually chosen to be 5 or 10.
• Advantages:
• Makes use of more data for training (data often scarce)
• Repeated estimates mitigates variance of splits
• Can create confidence intervals for validation error
• Disadvantages:
• More computationally expensive

(Demo)

Estimating Risk, Bias, and Variance

• CV lets us see bias and variance!
• Training errors show model bias
• Validation errors show risk, CIs show model variance

Break!

Fill out Attendance:

http://bit.ly/at-d100

Regularization

Weighty Issues

Large model weights create complicated models.

Idea: Prevent large weights to make simpler models.

Regularization

• Regularization (aka shrinkage) adds a penalty for model weights to the loss function.
• MSE loss with L2 regularization:

λ: Regularization parameter (non-negative)

Same ol’ loss as usual

Penalty for θ values

Ridge and Lasso Regression

• Ridge regression: linear model with L2 regularization

• Lasso regression: linear model with L1 regularization

L₂ norm

(Demo)

L₁ norm

Regularization Parameter

• λ is the regularization parameter.
• Higher values penalize model weights more.
• Discuss:
• What happens when λ = 0?
• What happens when λ = ∞?
• Does this change between L2 and L1 regularization?

What happens when...

• λ = 0?
• No regularization, back to linear model
• λ = ∞?
• Flat line, all model weights = 0
• Does this change between L2 and L1 regularization?
• No

Don’t regularize the bias

• Notice that we don’t regularize the bias term!

• Discuss: why not?
• Bias term doesn’t add complexity to model

Normalize Data Before Using Regularization

• Before using regularization, normalize data
• Subtract mean and scale data to lie between -1 and 1.
• Discuss: what happens if we don’t do this?
• Artificial penalty on features with small numbers

Exercise to take home:

• Prove that the stochastic gradient descent update rule for ridge regression is:

• (Lasso is a bit tricker but also doable.)

Why two kinds of regularization?

• Intuitive, hand-wavy explanation:
• L2 regularization typically has all non-zero weights.
• Makes sense when we think many small factors contribute to outcome.
• L1 regularization will set some model weights = 0 depending on how big λ is.
• L1 regularization lets us perform feature selection.
• Makes sense when we think a few major factors contribute to outcome.

A more sophisticated explanation

Suppose we have a linear model with two parameters and no intercept term.

As we tweak the two parameters, loss changes.

Without regularization, we just pick θ̂.

θ₁

θ₂

A more sophisticated explanation

Regularization balances loss with the regularization penalty.

For L2 regularization, we have circular contours for the penalty. Why?

θ₁

θ₂

A more sophisticated explanation

For L1 regularization, we have diamond-shaped contours for the penalty. Why?

Notice that this sets one parameter = 0!

This idea extends to multiple dimensions.

θ₁

θ₂

A tuning knob for bias-variance

• Regularization gives us yet another way to manage the bias-variance tradeoff.
• Increase λ = more bias, less variance
• Decrease λ = less bias, more variance
• How do we pick λ?
• Cross-validation!

Summary

• K-Fold cross-validation lets us estimate model bias, model variance, and overall risk.
• We use CV to perform model and feature selection.
• Regularization gives us a way to add complexity to our models while avoiding overfitting.
• We use CV to tune the regularization amount.