1 of 27

�

​

​

​

​

Swayam Prabha

Course Title

Multivariate Data Mining- Methods and Applications

Lecture 35

Regression Trees

By

Anoop Chaturvedi

Department of Statistics, University of Allahabad

Prayagraj (India)

​

Slides can be downloaded from https://sites.google.com/view/anoopchaturvedi/swayam-prabha

2 of 27

Example: Carseats data set ⇒ Simulated data set containing sales of child car seats at 400 different stores. Data frame with 400 observations on the following 11 variables.

Found in the ISLR package of R.

Objective ⇒ Predicting Sales based on the other variables in the data set.

Sales is a continuous variable and recoded as a binary variable.

Binary Sales Variable High ⇒ Yes if the Sales exceed 8, and No otherwise.

High is a binary variable and this is a classification problem and requires the use of a classification tree.

Data Mining_Anoop Chaturvedi

2

3 of 27

11 Variables

Sales: Unit sales (in thousands)

CompPrice: Price charged by competitor

Income: Community income level (thousands of dollars)

Advertising: Local advertising budget (in thousands of dollars)

Population: Population size in region (in thousands)

Price: Price company charges for car seats at each site

ShelveLoc: Quality of the shelving location for car seats Bad, Good and Medium

Age: Average age of local population

Education: Education level at each location

Urban: Store in an urban or rural location (No or Yes)

US: Whether the store is in the US or not (Yes or No)

Data Mining_Anoop Chaturvedi

3

4 of 27

Data Mining_Anoop Chaturvedi

4

5 of 27

Data Mining_Anoop Chaturvedi

5

6 of 27

.

Data Mining_Anoop Chaturvedi

6

Total seven splits

7 of 27

.

Data Mining_Anoop Chaturvedi

7

69% correctly classified, 31% misclassified

8 of 27

Data Mining_Anoop Chaturvedi

8

9 of 27

.

Data Mining_Anoop Chaturvedi

9

10 of 27

.

Data Mining_Anoop Chaturvedi

10

Use the Pruned Tree to Predict the Test Data and Evaluate Tree Performance

60.5% correctly classified, 39.5% misclassified

11 of 27

Regression trees: Key aspects

  • Recursive Partitioning ⇒ Partitioning features space into smaller regions. At each step, select a feature and a best-split point by minimizing the variance of the target variable within each subset. Iterate the process until it reaches a maximum depth, minimum number of samples in a node, or no further reduction in variance.
  • Splitting Strategy ⇒ Based on mean squared error (MSE)/ mean absolute error (MAE). Select the feature and split point that minimize the weighted sum of variances of the target variable in the resulting subsets.

Data Mining_Anoop Chaturvedi

11

12 of 27

  • Tree Pruning ⇒ As the tree grows deep and captures noise in the training data leading to overfitting. Use cost-complexity pruning or weakest link pruning to remove nodes that do not significantly improve predictive performance on a validation dataset.

Data Mining_Anoop Chaturvedi

12

13 of 27

  •  

Data Mining_Anoop Chaturvedi

13

14 of 27

  •  

Data Mining_Anoop Chaturvedi

14

15 of 27

  •  

Data Mining_Anoop Chaturvedi

15

16 of 27

  •  

Data Mining_Anoop Chaturvedi

16

17 of 27

  •  

Data Mining_Anoop Chaturvedi

17

18 of 27

  •  

Data Mining_Anoop Chaturvedi

18

19 of 27

Prediction ⇒ Involves drop-down new data to the root node traversing the tree to a leaf node based on the values of the input features.

The predicted value for a new data point is the constant value associated with the leaf node it ends up in.

Example: In Carseats data set, predict Sales based on the other variables. Because Sales is a continuous variable, this is a regression problem. Split data in training and test sets.

Regression tree contains 6 variables with 13 splits. Splits have occurred on the variables ShelveLoc, Price, Advertising, Income, Age, and CompPrice.

​

Data Mining_Anoop Chaturvedi

19

20 of 27

Data Mining_Anoop Chaturvedi

20

21 of 27

Data Mining_Anoop Chaturvedi

21

22 of 27

Data Mining_Anoop Chaturvedi

22

23 of 27

Data Mining_Anoop Chaturvedi

23

24 of 27

Data Mining_Anoop Chaturvedi

24

25 of 27

Data Mining_Anoop Chaturvedi

25

26 of 27

Data Mining_Anoop Chaturvedi

26

For unpruned regression tree

27 of 27

  • Regression trees can be easily visualized and interpreted. Each split in the tree represents a decision rule based on a feature and its threshold value.
  • Ensemble Methods ⇒ To improve predictive performance, regression trees can be combined into ensemble methods such as Random Forests, which aggregate the predictions of multiple trees to reduce overfitting and improve generalization.
  • Limitations ⇒ Regression trees have instability to small changes in the data. Also have difficulty in capturing complex relationships between features.

Data Mining_Anoop Chaturvedi

27