1 of 42

CSE 5523: �Regression

2 of 42

Hw-1

  • Will be released tonight (midnight)
    • You will receive Carmen’s announcement
  • Will be due on 9/17 (midnight)
  • Questions in both the programming set and the problem set
  • Please use Piazza or come to the office hours for discussion
  • Please start working on it ASAP

​

3 of 42

Today

​

Linear regression

  • Review
  • Linear regression (continued)
  • Ridge regression
  • A little more math

​

Nonlinear regression

​

4 of 42

Linear regression

  •  

4

 

 

 

5 of 42

Linear regression

  •  

5

x: area

y: price

Regression (house price):

From x (area), predict y (price)

Linear relationship

 

6 of 42

Today

​

Linear regression

  • Review
  • Linear regression (continued)
  • Ridge regression
  • A little more math

​

Nonlinear regression

​

7 of 42

Least mean square (LMS) solution

  •  

7

 

8 of 42

Least mean square (LMS) solution

  • Continued

​

8

 

 

9 of 42

Least mean square (LMS) solution

  •  

9

 

 

More useful tools in the Matrix Cookbook!

10 of 42

Least mean square (LMS) solution: another way

10

 

11 of 42

Mini summary

  •  

11

12 of 42

Today

​

Linear regression

  • Review
  • Linear regression (continued)
  • Ridge regression
  • A little more math

​

Nonlinear regression

​

13 of 42

Recap: Eigen-decomposition (ED)

  •  

13

 

14 of 42

Recap: Singular value decomposition (SVD)

  • Singular value decomposition (SVD)

​

14

 

 

(D x D)

(D x N)

(N x N)

 

Non-negative

Orthonormal, unitary

Orthonormal, unitary

(D x N)

15 of 42

Recap: SVD + ED

  •  

15

 

 

Definition of SVD

16 of 42

Why not invertible?

  • Look deeper!

​

​

​

​

    • Not invertible if there are zeros along the diagonal

​

​

16

 

17 of 42

How to make it invertible?

  •  

17

 

18 of 42

Practice!

  •  

18

19 of 42

Summary

  • Linear regression (linear curve fitting)
    • Model input-output relationships via linear functions
  • To learn regression model parameters
    • Given training data examples
    • Choose an error/loss function
  • Linear least mean square (LMS)
    • Minimize sum of residual sum of square (RSS)
      • Calculus and Linear Algebra
    • Ridge regression for noninvertible cases

19

20 of 42

Today

​

Linear regression

  • Review
  • Linear regression (continued)
  • Ridge regression
  • A little more math

​

Nonlinear regression

​

21 of 42

A little more math

  •  

21

22 of 42

A little more math

  •  

22

23 of 42

What does ridge regression do?

  •  

23

 

 

 

 

 

 

Ridge regression: The one closest to the origin

24 of 42

Questions?

25 of 42

A little more math: convex functions

  •  

25

 

 

 

 

 

 

 

 

 

26 of 42

A little more math: convex functions

  •  

26

 

 

 

 

 

 

 

 

 

27 of 42

A little more math: convex functions

  •  

27

 

 

28 of 42

Caution

  • When the function to be minimized is convex:
    • Any local optimum is a global optimum
    • However, it does not imply there exists closed-form solutions
    • Indeed, most of the (convex) optimization problems do not have closed-form solutions

29 of 42

Questions?

30 of 42

Today

​

Linear regression

  • Review
  • Linear regression (continued)
  • Ridge regression
  • A little more math

​

Nonlinear regression

​

31 of 42

Nonlinear regression

  •  

31

 

32 of 42

Nonlinear regression: representation

  •  

32

 

33 of 42

Nonlinear regression: representation

  •  

33

34 of 42

Nonlinear regression: representation

  •  

34

35 of 42

Nonlinear regression: parameter estimation

  •  

35

 

36 of 42

Questions?

37 of 42

Nonlinear regression: example

  •  

37

 

 

 

 

 

 

 

38 of 42

Nonlinear regression: example

  • Training data are generated from a sin function + noise
    • Under-fitting

h too simple

​

​

​

​

​

​

    • Over-fitting

h too complicated

training error = 0

38

[Slides: from USC CSCI567]

39 of 42

Nonlinear regression: example

  • Training data are generated from a sin function + noise
    • Just good enough

​

​

​

​

​

​

39

[Slides: from USC CSCI567]

40 of 42

Training vs. testing

  • Models learned from the training data should be applicable to test data

40

 

 

x

y

In training, we only see training data!

Choosing a more complicated hypothesis class does not necessarily lead to lower test errors!

41 of 42

Over-fitting vs. hypothesis class (brief)

  • Over-fitting:
    • Small training error
    • Large test error (even larger than some other “simpler” models)

​

  • How to quantify hypothesis class’ complexity?
    • KNN: smaller K, larger complexity
    • Polynomial regression: larger M, larger complexity

​

  • How to choose?

​

​

​

41

42 of 42

Nonlinear regression: example

  •  

42