1 of 29

Linear Regression

Fit a line to the data

Calculate

R²

Calculate F-Test

2 of 29

Fit a line to the data

Weight

Size

Weight

Size

3 of 29

Which one fits best?

Fit a line to the data

Weight

Size

Weight

Size

Weight

Size

Line 1

Line 2

Line 3

4 of 29

Weight

Size

Weight

Size

Weight

Size

Line 1

Line 2

Line 3

Residuals 1

Residuals 2

Residuals 3

The best line is the one that minimizes the sum of the squared residuals!

RSS 2 > RSS 1 > RSS 3

Fit a line to the data

Ordinary Least Squares

5 of 29

How to find the best line?

x

y

y = b + a*x

Fit a line to the data

What is a line?

What is a linear model?

Weight

Size

Parameters: b and a

6 of 29

How to find the best line?

Fit a line to the data

Weight

Size

sizeSS = size_fitSS + residualsSS

7 of 29

Fit a line to the data

RSS

It is time for derivatives!

8 of 29

Fit a line to the data

Linear Model in matrix notation

1

9 of 29

Fit a line to the data

Linear Model in matrix notation

10 of 29

Fit a line to the data

Parameters estimation

Equating to zero:

Ordinary Least Squares

11 of 29

Fit a line to the data

Parameters estimation

Maximum likelihood

Gaussian distribution

Multivariate Gaussian distribution

12 of 29

Fit a line to the data

Parameters estimation

Maximum likelihood

Weight

Size

Any Size value belongs to a Gaussian distribution, the mean is the fitted value.

13 of 29

Fit a line to the data

Parameters estimation

Maximum likelihood

14 of 29

Linear Regression

Calculate

R²

Calculate F-Test

Fit a line to the data

Ordinary Least Squares

Maximum likelihood

Derivatives

15 of 29

Calculate R²

How good is that prediction?

R²

16 of 29

Calculate R²

Size

Weight

17 of 29

Calculate R²

Size

Weight

Size

18 of 29

Calculate R²

Size

Weight

Size

19 of 29

Calculate R²

Size

Weight

Size

SS(mean) = 1080

Var(mean) = 12

SS(fit) = 720

Var(fit) = 4

20 of 29

Calculate R²

Size

Weight

Size

SS(mean) = 1080

Var(mean) = 12

SS(mean) = 120

Var(fit) = 1.34

21 of 29

Calculate R²

Weight

Size

Water consumption

Size

Fur Softness

Size

SS(mean) = equal for all

SS(fit) = different for each

SS(fit3) > SS(fit2) > SS(fit1)

R²(fit1) > R²(fit2) > R²(fit3)

22 of 29

Calculate R²

Pearson correlation coefficient

r

R²

2

23 of 29

Linear Regression

Calculate F-Test

Fit a line to the data

Ordinary Least Squares

Maximum likelihood

Derivatives

Calculate

R²

r

R²

2

Person’s correlation

24 of 29

Calculate F-Test

Hypotheses Testing

Which model (reduced or full) does a better job?

SSE(R) can never be smaller than SSE(F)

If SSE(R) is close to SSE(F): new parameter reduces little of the variation

If SSE(R) and SSE(F) differ greatly: new parameter substantially reduces the variance

How different does SSE(R) have to be from SSE(F) in order to justify using the full model?

25 of 29

Calculate F-Test

Transforming into variances

26 of 29

Calculate F-Test

The F distribution

Weight

Size

Weight

Size

Generate sets of random data (n=9)

Calculate SS(mean) and SS(fit)

Weight

Size

Weight

Size

Weight

Size

Weight

Size

Calculate F

F = 2

F = 3

F = 4

Plot the results in a histogram

F

Frequency

27 of 29

Calculate F-Test

The F distribution

The F distribution depends on two parameters (d1 and d2), which are the degrees of freedom of the numerator and denominator.

F

Frequency

Numerator = (9-1) – (9-2) = 8 – 7 = 1

Denominator = (9-2) = 7

Shiny

28 of 29

Calculate F-Test

p-value for F-distribution

In this situation, a model with one predictor variable and 9 samples would need a F value higher than 5.59 to show a p-value < 0.05 and the null hypothesis would be rejected.

29 of 29

Linear Regression

Fit a line to the data

Ordinary Least Squares

Maximum likelihood

Derivatives

Calculate

R²

r

R²

2

Pearson’s correlation

Calculate F-Test