1 of 57

Join at slido.com�#2060756

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2060756

2 of 57

Linear Regression (1)

Lecture 6

Model, error function, normal equations, geometry, evaluation, and regularization

EECS 189/289, Fall 2026 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

3 of 57

Fitting a Straight Line to Non-Linear Data

2060756

4 of 57

Roadmap

  • Linear Regression
  • Basis Functions
  • Vectorizing Calculations
  • Error Function
  • Error Function Minimization
  • Geometric Interpretation
  • Evaluation

2060756

5 of 57

Linear Regression

  • Linear Regression
  • Basis Functions
  • Vectorizing Calculations
  • Error Function
  • Error Function Minimization
  • Geometric Interpretation
  • Evaluation

2060756

6 of 57

Linear Regression Outline

L

LEARNING PROBLEM

M

MODEL DESIGN

O

OPTIMIZATION

P

PREDICT & EVALUATE

L

M

P

O

Supervised learning of scalar target values

 

 

 

2060756

7 of 57

Learning Problem

L

LEARNING PROBLEM

L

Supervised learning of scalar target values

2060756

8 of 57

Regression

  •  

 

 

Domain

Model

 

 

 

2060756

9 of 57

Model Design

L

LEARNING PROBLEM

M

MODEL DESIGN

L

M

Supervised learning of scalar target values

 

2060756

10 of 57

Supervised Linear Regression

  •  

 

2060756

11 of 57

The Simplest Linear Regression Model

  •  

Slope (rate of change)

Intercept (shift)

Predicted output

 

2060756

12 of 57

The Simplest Linear Regression Model

  •  

Predicted output

 

Interactive link

2060756

13 of 57

Which of the following is a linear regression model?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2060756

14 of 57

Basis Functions

  • Linear Regression
  • Basis Functions
  • Vectorizing Calculations
  • Error Function
  • Error Function Minimization
  • Geometric Interpretation
  • Evaluation

2060756

15 of 57

Linear Functions From Slido

These are all linear models with different basis functions.

​

We will now see what basis functions are…

2060756

16 of 57

What Does It Mean To Be a Linear Model?

In what sense are the previous plots linearly modeled?

​

​

Are linear models linear in the

  1. Features?
  2. Parameters?

 

2060756

17 of 57

Are linear models linear in the ...

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2060756

18 of 57

What Does It Mean To Be a Linear Model?

In what sense are the previous plots linearly modeled?

​

​

Are linear models linear in the

  1. Features?
  2. Parameters?

 

Feature Functions

Linear in the Parameters

2060756

19 of 57

Basis Functions

  •  

 

2060756

20 of 57

More on Basis Function

2060756

21 of 57

Comparing Basis Function Families

 

2060756

22 of 57

Vectorizing Calculations

  • Linear Regression
  • Basis Functions
  • Vectorizing Calculations
  • Error Function
  • Error Function Minimization
  • Geometric Interpretation
  • Evaluation

2060756

23 of 57

Vectorizing Calculations

  •  

 

 

 

 

2060756

24 of 57

Vectorizing Calculations

 

 

 

 

2060756

25 of 57

Matrix Notation

  •  

 

 

 

 

 

2060756

26 of 57

Matrix Notation

 

 

 

 

 

 

 

 

 

 

 

Design matrix

2060756

27 of 57

Error Function

  • Linear Regression
  • Basis Functions
  • Vectorizing Calculations
  • Error Function
  • Error Function Minimization
  • Geometric Interpretation
  • Evaluation

2060756

28 of 57

Optimization

L

LEARNING PROBLEM

M

MODEL DESIGN

O

OPTIMIZATION

L

M

O

Supervised learning of scalar target values

 

 

2060756

29 of 57

Error Function Visualization

 

2060756

30 of 57

Error Function

  •  

 

  • Non-negative quantity.
  • Only 0 if all predictions are equal to targets.

2060756

31 of 57

Error Function Visualization

 

2060756

32 of 57

Error Function Minimization

  • Linear Regression
  • Basis Functions
  • Vectorizing Calculations
  • Error Function
  • Error Function Minimization
  • Geometric Interpretation
  • Evaluation

2060756

33 of 57

Error Function Minimization

  •  

 

 

 

Finding the optimum solution

 

2060756

34 of 57

Error Function Minimization

 

 

 

 

Separating the terms

 

2060756

35 of 57

Error Function Minimization

 

 

 

 

 

 

 

 

 

 

Separating the terms

2060756

36 of 57

Error Function Minimization

 

 

 

 

 

 

 

 

 

 

Separating the terms

2060756

37 of 57

Error Function Minimization

 

 

 

 

 

 

 

Reordering

 

2060756

38 of 57

Error Function Minimization

 

 

 

 

 

 

 

 

Takeaway

 

Normal equations for the least squares problem

 

2060756

39 of 57

Geometric Interpretation

  • Linear Regression
  • Basis Functions
  • Vectorizing Calculations
  • Error Function
  • Error Function Minimization
  • Geometric Interpretation
  • Evaluation

2060756

40 of 57

[Linear Algebra] Span

  •  

 

 

 

2060756

41 of 57

[Linear Algebra] Matrix-Vector Multiplication

  •  

 

=

 

 

 

…

 

 

 

=

 

 

…

 

=

 

 

+

 

 

+

 

 

+

…

2060756

42 of 57

Prediction Is a Linear Combination of Columns

 

 

 

 

 

 

2060756

43 of 57

What’s the geometry word for ‘closest point in a subspace’?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2060756

44 of 57

 

 

 

 

 

 

 

 

 

 

Length of the residual vector

2060756

45 of 57

Geometry of Least Squares in Plotly

Interactive link

2060756

46 of 57

[Linear Algebra] Orthogonality

  •  

 

 

 

 

 

…

 

 

 

 

…

 

=

 

 

 

…

 

 

…

 

 

2060756

47 of 57

Going Back to Our Error Function

 

 

 

Adding the definition of residual

 

 

 

Moving terms

 

Normal Equation

 

 

2060756

48 of 57

Evaluation

  • Linear Regression
  • Basis Functions
  • Vectorizing Calculations
  • Error Function
  • Error Function Minimization
  • Geometric Interpretation
  • Evaluation

2060756

49 of 57

Predict and Evaluate

L

LEARNING PROBLEM

M

MODEL DESIGN

O

OPTIMIZATION

P

PREDICT & EVALUATE

L

M

P

O

Supervised learning of scalar target values

 

 

 

2060756

50 of 57

Evaluation - Visualization

 

2060756

51 of 57

Evaluation - Metrics

​

​

​

​

​

​

​

​

​

​

2060756

52 of 57

Evaluation - Metrics

Mean Squared Error (MSE)

​

​

​

​

​

​

​

​

​

2060756

53 of 57

Evaluation - Metrics

Mean Squared Error (MSE)

​

Root Mean Squared Error (RMSE)

​

​

​

​

​

​

​

Moves the metric back to the original unit of the data compared to MSE

2060756

54 of 57

Visualizing the Sum of Squared Error of Regression Model

54

 

 

Goal of regression: Make the total area of the boxes as small as possible.

 

2060756

55 of 57

Visualizing the Sum of Squared Error of Intercept Model

55

 

 

 

2060756

56 of 57

R2: Quality of the Fit Relative to Intercept Model

56

 

 

 

 

unitless and only compares performance relative to mean baseline

2060756

57 of 57

Linear Regression (1)

Lecture 6

Credit: Joseph E. Gonzalez and Narges Norouzi

Reference Book Chapters: Chapter 1.2, Chapter 4.[1.4-1.6]