Regression 1

Assumption: Linear Model

2

Assumption: Linear Model

3

Linear Regression

4

Linear Regression as Optimization

5

Re-cast Problem as Least Squares

6

Optimization

7

Optimization: Note

8

the same principle in a higher dimension

Revisit: Least-Square Solution

9

1. Solve using Linear Algebra

• known as least square

​

10

1. Solve using Linear Algebra

• known as least square

​

11

2. Solve using Gradient Descent

12

3. Solve using CVXPY Optimization

13

3. Solve using CVXPY Optimization

14

15

Regression with Outliers

16

Regression with Outliers

17

18

Think About What Makes Different

• It is important to understand what makes them different
• Sensitivity to outliers in L2 norm
• Robustness of L1 norm

​

​

​

​

​

​

​

​

​

• Squaring the residuals magnifies the effect of large errors, meaning that outliers have a significant influence on the final regression line

​

​

19

Scikit-Learn

• Machine Learning in Python
• Simple and efficient tools for data mining and data analysis
• Accessible to everybody, and reusable in various contexts
• Built on NumPy, SciPy, and matplotlib
• Open source, commercially usable - BSD license
• https://scikit-learn.org/stable/index.html#

​

20

Scikit-Learn

21

Scikit-Learn: Regression

22

Scikit-Learn: Regression

23

Multivariate Linear Regression

24

Multivariate Linear Regression

25

Multivariate Linear Regression

26

​

​

​

• Some features may have little impact on the target variable.
• If a feature’s coefficient is relatively small compared to others, it contributes minimally to the prediction and can be eliminated to simplify the model.
• Simplifying the model leads to better generalization and computational efficiency.

​

• Before feature selection

​

• After feature selection

27

Nonlinear Regression

• Explores how to construct a regression model capable of approximating non-linearly distributed data

28

Nonlinear Regression with Polynomial Features

• Polynomial (here, quad is used as an example)

29

Nonlinear Regression with Polynomial Features

• Polynomial (here, quad is used as an example)

30

Polynomial Regression

31

Nonlinear Regression (Actually Linear Regression)

32

Linear Basis Function Model

33

Recap: Nonlinear Regression

• Polynomial (here, quad is used as an example)

34

Different perspective:

- Approximate a target function as a linear combination of basis

Construct Explicit Feature Vectors

• Consider linear combinations of fixed nonlinear functions
• Polynomial
• Radial Basis Function (RBF)

35

Polynomial Basis

1) Polynomial functions

36

Nonlinear Function with Polynomial Basis

37

RBF Basis

38

Nonlinear Function with RBF Basis

39

Nonlinear Regression with Linear Basis Function Models

• Polynomial functions

• RBF functions

​

40

Regression 2

Linear Regression: Advanced

42

• Overfitting
• Regularization (Ridge and Lasso)

Overfitting: Start with Linear Regression

43

Recap: Nonlinear Regression

• Polynomial (here, quad is used as an example)

44

Nonlinear Regression

45

10 input points with degree 9 (or 10)

Polynomial Fitting with Different Degrees

46

Low error on input data points,

but high error nearby

• Underfit
• Good fit
• Overfit

​

Important to find the right balance between model complexity and generalization

Loss

• Loss: Residual Sum of Squares (RSS)
• decreases as the polynomial degree increases
• a lower error does not necessarily indicate a better model

47

Low error on input data points,

but high error nearby

Issue with Rich Representation

• Low error on input data points, but high error nearby
• Low error on training data, but high error on testing data

48

Overfitting

• One of the most common problem data science professionals face

​

• Overfitting occurs when a machine learning model performs well on the training data but poorly on unseen or test data.

​

• The model essentially "memorizes" the noise and details in the training set rather than learning the general patterns that can be applied to new data.

49

Signs of Overfitting

• Large gap between training and test loss:

​

• Training loss continues to decrease, but test loss increases.

50

Causes of Overfitting

• Model Complexity
• A model with too many parameters (e.g., deep neural networks) can overfit the training data.

​

• Insufficient Training Data
• If the training dataset is small, the model may fit every point precisely, capturing noise rather than general patterns.

​

• Noisy Data
• If the data has a lot of noise or irrelevant features, the model may try to learn this noise.

​

• Too Many Training Iterations
• Training for too long can lead to overfitting, where the model starts fitting noise instead of actual patterns.

​

51

Generalization Error

• the difference between a machine learning model's performance on the training data and its performance on unseen data (test or validation data)

52

Regularization to Reduce Overfitting

53

Generalization Error

54

Representational Difficulties

55

With Less Basis Functions: Fewer RBF Centers

56

With Less Basis Functions: Fewer RBF Centers

• Least-squares fits for different numbers of RBFs

​

57

Representational Difficulties

58

Regularization (Shrinkage Methods)

• Often, overfitting associated with very large estimated parameters
• We want to balance
• how well function fits data
• magnitude of coefficients 

​

​

​

​

​

​

​

• multi-objective optimization
• 𝜆 is a tuning parameter

59

Regularization (Shrinkage Methods)

60

RBF: Start from Rich Representation

61

RBF with Regularization

62

RBF with Regularization

63

64

How L2 Regularization Works

​

​

​

​

• Effect on Weights:
• L2 regularization forces the model to keep weights small.
• Interpretation:
• L2 regularization distributes the penalty uniformly across all weights, shrinking their magnitude.
• Overfitting Reduction:
• By limiting the model's ability to learn large weights, the model becomes less likely to overfit the noise in the training data.

​

​

65

66

RBF with LASSO

​

​

​

​

​

​

​

​

• Approximated function looks similar to that of ridge regression

67

68

LASSO

Ridge

• 'Shrink' some coefficients exactly to 0
• knock out certain features
• the L1 penalty has the effect of forcing some of the coefficient estimates to be exactly zero

​

• Non-zero coefficients indicate the 'selected' features

69

LASSO

Sparsity and Feature Selection

• The L1 penalty in LASSO encourages sparsity in the coefficient estimates, meaning it drives some coefficients to be exactly zero.

​

• Useful for feature selection, as it allows the model to identify and retain only the most relevant predictors, simplifying the model and potentially improving interpretability.

70

Regression with Selected Features

71

LASSO vs. Ridge

• Equivalent optimization formulations for both Ridge and LASSO

​

72

LASSO vs. Ridge

• Equivalent optimization formulations for both Ridge and LASSO

​

73

Evaluation

• Adding more features will always decrease the loss
• How do we determine when an algorithm achieves “good” performance?

​

• A better criterion:
• Training set (e.g., 70 %)
• Testing set (e.g., 30 %)

​

​

​

​

​

​

• Performance on testing set called generalization performance

​

74

Regression 3

Linear Regression Examples

76

• De-noising
• Total Variation

De-noising Signal

77

Transform it to an Optimization Problem

78

Source:

• Boyd & Vandenberghe's book "Convex Optimization“
• http://cvxr.com/cvx/examples/ (Figures 6.8-6.10: Quadratic smoothing)
• Week 4 of Linear and Integer Programming by Coursera of Univ. of Colorado

Transform it to an Optimization Problem

79

Least-Square Problems

80

Coded in Python

81

Notes

• While this process highlights the potential of regression for noise reduction, it is important to note that such tasks are typically achieved more effectively and efficiently using low-pass filters in either the time or frequency domain.

​

• Nevertheless, this example serves to highlight the broader applicability of regression methods, demonstrating their potential to address various practical problems.

​

82

83

CVXPY Implementation

84

85

86

87

88

Signal with Sharp Transition + Noise

89

Chapter 6.3 from Boyd & Vandenberghe's book "Convex Optimization”

​

​

​

​

​

• Quadratic smoothing smooths out both noise and sharp transitions in signal, but this is not what we want
• We will not be able to preserve the signal’s sharp transitions.

​

• Any ideas ?

​

90

91

​

​

​

​

​

​

​

• Total Variation (TV) smoothing preserves sharp transitions in signal, and this is not bad

​

• Note that how TV reconstruction does a better job of preserving the sharp transitions in the signal while removing the noise.

​

92

Total Variation Image

93

Idea comes from http://www2.compute.dtu.dk/~pcha/mxTV/

Total Variation Image

94

Idea comes from http://www2.compute.dtu.dk/~pcha/mxTV/

Summary

• Thses examples demonstrates how regression principles extend beyond conventional predictive models and can be applied effectively to complex tasks such as image denoising and signal reconstruction.

95