1 of 23

�

​

​

​

​

Data Mining_Anoop Chaturvedi

1

Swayam Prabha

Course Title

Multivariate Data Mining- Methods and Applications

Lecture 10

Shrinkage Estimation

By

Anoop Chaturvedi

Department of Statistics, University of Allahabad

Prayagraj (India)

​

Slides can be downloaded from https://sites.google.com/view/anoopchaturvedi/swayam-prabha

2 of 23

Variable Selection Procedures⇒

  • Variables are retained or discarded.
  • Often lead to high variance and prediction error.�Shrinkage Methods
  • More continuous and don’t suffer much from high variability.�Penalized Regression Estimators
  • Shrink the estimators by imposing a penalty on their size.�Sparsity: Large number of predictors recorded but a relatively small number/ proportion have strong effects.

Data Mining_Anoop Chaturvedi

2

3 of 23

 

Data Mining_Anoop Chaturvedi

3

4 of 23

 

Data Mining_Anoop Chaturvedi

4

5 of 23

  •  

Data Mining_Anoop Chaturvedi

5

6 of 23

 

Data Mining_Anoop Chaturvedi

6

7 of 23

 

Econometrics_Anoop Chaturvedi

7

8 of 23

�

Econometrics_Anoop Chaturvedi

8

​

9 of 23

 

Econometrics_Anoop Chaturvedi

9

10 of 23

Figure: Univariate ridge trace and VIF trace plots for the coefficients.

Econometrics_Anoop Chaturvedi

10

Biasing parameter c

Biasing parameter c

11 of 23

  •  

Econometrics_Anoop Chaturvedi

12 of 23

  •  

Data Mining_Anoop Chaturvedi

12

13 of 23

  •  

Data Mining_Anoop Chaturvedi

13

14 of 23

Example: Data ⇒ mtcars ⇒ glmnet package in R

Input variables ⇒ mpg, wt, drat, qsec,

Output variable ⇒ hp

Full Model:

​

​

​

​

​

Train set: 22, Test: 10�

​

​

Data Mining_Anoop Chaturvedi

14

 

Estimate

Std. Error

t value

Pr(>|t|)

Intercept

473.779

105.213

4.503

0.000116

mpg

2.877

2.381

1.209

0.237319

wt

26.037

13.514

1.927

0.064600 .

drat

4.819

15.952

0.302

0.764910

qsec

-20.751

3.993

5.197

1.79 e-05

15 of 23

  •  

Data Mining_Anoop Chaturvedi

15

 

16 of 23

 

Data Mining_Anoop Chaturvedi

16

17 of 23

�

Data Mining_Anoop Chaturvedi

17

 

18 of 23

 

Data Mining_Anoop Chaturvedi

18

19 of 23

 

Data Mining_Anoop Chaturvedi

19

20 of 23

������

Constraint regions for Ridge and LASSO for two parameter case

​

Ridge

​

​

​

LASSO

 

Data Mining_Anoop Chaturvedi

20

 

 

21 of 23

  •  

Data Mining_Anoop Chaturvedi

21

22 of 23

  •  

Data Mining_Anoop Chaturvedi

22

23 of 23

 

Data Mining_Anoop Chaturvedi

23