GMM Recap + Linear Regression (1)
Lecture 6
Getting started with linear regression
EECS 189/289, Fall 2025 @ UC Berkeley
Joseph E. Gonzalez and Narges Norouzi
EECS 189/289, Fall 2025 @ UC Berkeley
Joseph E. Gonzalez and Narges Norouzi
Join at slido.com�#3689302
The Slido app must be installed on every computer you’re presenting from
Do not edit�How to change the design
3689302
Roadmap
3689302
Gaussian Mixture Model
3689302
Gaussian Mixture Model (GMM)
3689302
Gaussian Mixture Model (GMM)
3689302
Demo
Gaussian Mixture Model
3689302
The GMM is a Latent Variable Model
3689302
The GMM is a Generative Model
3689302
Demo
Sampling from a GMM
3689302
Latent Variable Posteriors
Should these points be red or blue�or both?
3689302
Estimating the GMM Parameters
3689302
Quick Recap
If we knew the model parameters, we could easily compute the cluster assignments.
If we knew the cluster assignments, we could easily estimate the model parameters.
How can we solve this cyclic dependency?
Model Parameters
Cluster Assignments
3689302
The EM Algorithm
Easy to optimize joint probability
Current distribution �over the latent Z
Updates distribution over Z.
3689302
The EM Algorithm: E-step
N
K
3689302
Lagrangian for the �normalization constraint
3689302
3689302
3689302
The EM Algorithm for GMMs
3689302
Demo
Implementing EM
for GMMs
3689302
Linear Regression
3689302
Linear Regression Outline
L
LEARNING PROBLEM
M
MODEL DESIGN
O
OPTIMIZATION
P
PREDICT & EVALUATE
L
M
P
O
Supervised learning of scalar target values
3689302
Learning Problem
L
LEARNING PROBLEM
L
Supervised learning of scalar target values
3689302
Regression
Domain
Model
3689302
Model Design
L
LEARNING PROBLEM
M
MODEL DESIGN
L
M
Supervised learning of scalar target values
3689302
Supervised Linear Regression
3689302
The Simplest Linear Regression Model
Slope (rate of change)
Intercept (shift)
Predicted output
3689302
The Simplest Linear Regression Model
Predicted output
3689302
Which of the following is a linear regression model?
The Slido app must be installed on every computer you’re presenting from
Do not edit�How to change the design
3689302
Basis Functions
3689302
Linear Functions From Slido
These are all linear models with different basis functions.
We will now see what basis functions are…
3689302
What Does It Mean To Be a Linear Model?
In what sense are the previous plots linearly modeled?
Are linear models linear in the
3689302
Are linear models linear in the
The Slido app must be installed on every computer you’re presenting from
Do not edit�How to change the design
3689302
What Does It Mean To Be a Linear Model?
In what sense are the previous plots linearly modeled?
Are linear models linear in the
Feature Functions
Linear in the Parameters
3689302
Basis Functions
3689302
More on Basis Function
3689302
Basis Functions as Features
3689302
How Would Basis Functions Improve Predictions?
Using polynomial basis functions of degree 5, we redefined the linear regression equation as:
Looking at the Mean Squared Error (MSE) between targets and predictions, the polynomial fit has a better performance.
3689302
Vectorizing Calculations
3689302
Vectorizing Calculations
3689302
Vectorizing Calculations
3689302
Matrix Notation
3689302
Matrix Notation
Design matrix
3689302
Error Function
3689302
Optimization
L
LEARNING PROBLEM
M
MODEL DESIGN
O
OPTIMIZATION
L
M
O
Supervised learning of scalar target values
3689302
Error Function
3689302
Error Function Visualization
3689302
Error Function Minimization
3689302
Error Function Minimization
Finding the optimum solution
3689302
Error Function Minimization
Separating the terms
3689302
Error Function Minimization
Reordering
3689302
Error Function Minimization
Takeaway
Normal equations for the least squares problem
3689302
Solving Normal Equation Runtime
Time complexity
3689302
Geometric Interpretation
3689302
[Linear Algebra] Span
3689302
[Linear Algebra] Matrix-Vector Multiplication
=
…
=
…
=
+
+
+
…
3689302
Prediction Is a Linear Combination of Columns
3689302
What’s the geometry word for ‘closest point in a subspace’?
The Slido app must be installed on every computer you’re presenting from
Do not edit�How to change the design
3689302
Length of the residual vector
3689302
Geometry of Least Squares in Plotly
3689302
[Linear Algebra] Orthogonality
…
…
=
…
…
We will use this shortly
3689302
Going Back to Our Error Function
Adding the definition of residual
Moving terms
Normal Equation
3689302
Evaluation
3689302
Predict and Evaluate
L
LEARNING PROBLEM
M
MODEL DESIGN
O
OPTIMIZATION
P
PREDICT & EVALUATE
L
M
P
O
Supervised learning of scalar target values
3689302
Evaluation - Visualization
3689302
When you see a fan shape in the residual plot, what comes to mind?
The Slido app must be installed on every computer you’re presenting from
Do not edit�How to change the design
3689302
Evaluation - Metrics
| |
| |
| |
| |
| |
3689302
Evaluation - Metrics
Mean Squared Error (MSE) | |
| |
| |
| |
| |
3689302
Evaluation - Metrics
Mean Squared Error (MSE) | |
Root Mean Squared Error (RMSE) | |
| |
| |
| |
Moves the metric back to the original unit of the data compared to MSE
3689302
Evaluation - Metrics
Mean Squared Error (MSE) | |
Root Mean Squared Error (RMSE) | |
R-Squared (R2) Score | |
| |
| |
3689302
Visualizing the Sum of Squared Error of Regression Model
71
Goal of regression: Make the total area of the boxes as small as possible.
3689302
Visualizing the Sum of Squared Error of Intercept Model
72
3689302
R2: Quality of the Fit Relative to Intercept Model
73
unitless and only compares performance relative to mean baseline.
3689302
Evaluation - Metrics
Mean Squared Error (MSE) | |
Root Mean Squared Error (RMSE) | |
R-Squared (R2) Score | |
Mean Absolute Error (MAE) | |
| |
In the same unit as the data; similar to MSE but differs in how the penalization applies.
3689302
Evaluation - Metrics
Mean Squared Error (MSE) | |
Root Mean Squared Error (RMSE) | |
R-Squared (R2) Score | |
Mean Absolute Error (MAE) | |
Mean Absolute Percentage Error (MAPE) | |
3689302
Linear Regression (1)
Lecture 6
Credit: Joseph E. Gonzalez and Narges Norouzi
Reference Book Chapters: Chapter 1.[2.1-2.3], Chapter 4.[1.1, 1.4]