1 of 66

Sensitivity Analysis�

Dr. Debasis Samanta

Associate Professor

Department of Computer Science & Engineering

2 of 66

Topics in this session…

  • Introduction

​

  • Estimation Strategies

​

  • Accuracy Estimation

​

  • Error Estimation

​

  • Statistical Estimation

​

  • Performance Estimation

​

  • ROC Curve

​

BDSET Course

2

DSamanta@IIT Kharagpur

3 of 66

Introduction

  • A classifier is used to predict an outcome of a test data

​

    • Such a prediction is useful in many applications
      • Business forecasting, cause-and-effect analysis, etc.

​

    • A number of classifiers have been evolved to support the activities.
      • Each has their own merits and demerits

​

  • There is a need to estimate the accuracy and performance of the classifier with respect to few controlling parameters in data sensitivity

​

  • As a task of sensitivity analysis, we have to focus on

​

    • Estimation strategy
    • Metrics for measuring accuracy
    • Metrics for measuring performance

BDSET Course

3

DSamanta@IIT Kharagpur

4 of 66

BDSET Course

4

DSamanta@IIT Kharagpur

Estimation Strategy

5 of 66

Planning for Estimation

  • Using some “training data”, building a classifier based on certain principle is called “learning a classifier”.

​

  • After building a classifier and before using it for classification of unseen instance, we have to validate it using some “test data”.

​

  • Usually training data and test data are outsourced from a large pool of data already available.

​

​

​

​

split

​

​

Data set Estimation

​

BDSET Course

5

DSamanta@IIT Kharagpur

Training data

Test data

Learning technique

CLASSIFIER

6 of 66

Estimation Strategies

  • Accuracy and performance measurement should follow a strategy. As the topic is important, many strategies have been advocated so far. Most widely used strategies are

​

    • Holdout method

​

    • Random subsampling

​

    • Cross-validation

​

    • Bootstrap approach

​

BDSET Course

6

DSamanta@IIT Kharagpur

7 of 66

Holdout Method

  • This is a basic concept of estimating a prediction.

​

    • Given a dataset, it is partitioned into two disjoint sets called training set and testing set.

​

    • Classifier is learned based on the training set and get evaluated with testing set.

​

    • Proportion of training and testing sets is at the discretion of analyst; typically 1:1 or 2:1, and there is a trade-off between these sizes of these two sets.

​

    • If the training set is too large, then model may be good enough, but estimation may be less reliable due to small testing set and vice-versa.

BDSET Course

7

DSamanta@IIT Kharagpur

8 of 66

Random Subsampling

​

    • It is a variation of Holdout method to overcome the drawback of over-presenting a class in one set thus under-presenting it in the other set and vice-versa.

​

    • In this method, Holdout method is repeated k times, and in each time, two disjoint sets are chosen at random with a predefined sizes.

​

    • Overall estimation is taken as the average of estimations obtained from each iteration.

BDSET Course

8

DSamanta@IIT Kharagpur

9 of 66

Cross-Validation

  • The main drawback of Random subsampling is, it does not have control over the number of times each tuple is used for training and testing.

​

  • Cross-validation is proposed to overcome this problem.

​

  • There are two variations in the cross-validation method.

​

    • k-fold cross-validation

​

    • N-fold cross-validation

​

BDSET Course

9

DSamanta@IIT Kharagpur

10 of 66

k-fold Cross-Validation

  •  

BDSET Course

10

DSamanta@IIT Kharagpur

Learning technique

CLASSIFIER

D1

Di

Dk

Data set

Fold 1

Fold i

Fold k

Accuracy

Performance

11 of 66

N-fold Cross-Validation

  •  

BDSET Course

11

DSamanta@IIT Kharagpur

12 of 66

N-fold Cross-Validation : Issue

  • So far the estimation of accuracy and performance of a classifier model is concerned, the N-fold cross-validation is comparable to the others we have just discussed.

​

  • The drawback of N-fold cross validation strategy is that it is computationally expensive, as here we have to repeat the run N times; this is practically infeasible when data set is large.

​

  • In practice, the method is extremely beneficial with very small data set only, where as much data as possible to need to be used to train a classifier.

BDSET Course

12

DSamanta@IIT Kharagpur

13 of 66

Bootstrap Method

  • The Bootstrap method is a variation of repeated version of Random sampling method.

​

  • The method suggests the sampling of training records with replacement.

​

    • Each time a record is selected for training set, is put back into the original pool of records, so that it is equally likely to be redrawn in the next run.

​

    • In other words, the Bootstrap method samples the given data set uniformly with replacement.

​

  • The rational of having this strategy is that let some records be occur more than once in the samples of both training as well as testing.

​

    • What is the probability that a record will be selected more than once?

​

BDSET Course

13

DSamanta@IIT Kharagpur

14 of 66

Bootstrap Method

  •  

BDSET Course

14

DSamanta@IIT Kharagpur

15 of 66

Bootstrap Method : Implication

  • This is why, the Bootstrap method is also known as 0.632 bootstrap method

BDSET Course

15

DSamanta@IIT Kharagpur

16 of 66

BDSET Course

16

DSamanta@IIT Kharagpur

Accuracy Estimation

17 of 66

Accuracy Estimation

  •  

BDSET Course

17

DSamanta@IIT Kharagpur

18 of 66

Accuracy : True and Predictive

  •  

BDSET Course

18

DSamanta@IIT Kharagpur

19 of 66

Predictive Accuracy

Example 11.1 : Universality of predictive accuracy

​

  • Consider a classifier model MD developed with a training set D using an algorithm M.

​

  • Two predictive accuracies when MD is estimated with two different training sets T1 and T2 are

(MD)T1 = 95%

(MD)T2 = 70%

​

  • Further, assume the size of T1 and T2 are

|T1| = 100 records

|T2| = 5000 records.

​

  • Based on the above mentioned estimations, neither estimation is acceptable beyond doubt.

BDSET Course

19

DSamanta@IIT Kharagpur

20 of 66

Predictive Accuracy

  • With the above-mentioned issue in mind, researchers have proposed two heuristic measures

​

    • Error estimation using Loss Functions

​

    • Statistical Estimation using Confidence Level

​

  • In the next few slides, we will discus about the two estimations

BDSET Course

20

DSamanta@IIT Kharagpur

21 of 66

Error Estimation using Loss Functions

  •  

BDSET Course

21

DSamanta@IIT Kharagpur

N×(n+1)

22 of 66

Error Estimation using Loss Functions

  •  

BDSET Course

22

DSamanta@IIT Kharagpur

23 of 66

Statistical Estimation using Confidence Level

  •  

BDSET Course

23

DSamanta@IIT Kharagpur

N=10

N=50

N=100

N=250

N=500

N=1000

H

T

H

T

H

T

H

T

H

T

H

T

3

7

29

21

54

46

135

115

241

259

490

510

0.30

0.70

0.58

0.42

0.54

0.46

0.54

0.46

0.48

0.42

0.49

0.51

24 of 66

Statistical Estimation using Confidence Level

  •  

BDSET Course

24

DSamanta@IIT Kharagpur

25 of 66

Statistical Estimation using Confidence Level

  •  

BDSET Course

25

DSamanta@IIT Kharagpur

 

 

26 of 66

Statistical Estimation using Confidence Level

  •  

BDSET Course

26

DSamanta@IIT Kharagpur

​

0.5

0.7

0.8

0.9

0.95

0.98

0.99

​

0.67

1.04

1.28

1.65

1.96

2.33

2.58

 

27 of 66

Statistical Estimation using Confidence Level

  •  

BDSET Course

27

DSamanta@IIT Kharagpur

28 of 66

Statistical Estimation using Confidence Level

  •  

BDSET Course

28

DSamanta@IIT Kharagpur

29 of 66

BDSET Course

29

DSamanta@IIT Kharagpur

Performance Estimation

30 of 66

Performance Estimation of a Classifier

  • Predictive accuracy works fine, when the classes are balanced
    • That is, every class in the data set are equally important

​

  • In fact, data sets with imbalanced class distributions are quite common in many real life applications

​

  • When the classifier classified a test data set with imbalanced class distributions then, predictive accuracy on its own is not a reliable indicator of a classifier’s effectiveness.

​

Example 11.3: Effectiveness of Predictive Accuracy

    • Given a data set of stock markets, we are to classify them as “good” and “worst”. Suppose, in the data set, out of 100 entries, 98 belong to “good” class and only 2 are in “worst” class.

​

      • With this data set, if classifier’s predictive accuracy is 0.98, a very high value!
        • Here, there is a high chance that 2 “worst” stock markets may incorrectly be classified as “good”

​

      • On the other hand, if the predictive accuracy is 0.02, then none of the stock markets may be classified as “good”

BDSET Course

30

DSamanta@IIT Kharagpur

31 of 66

Performance Estimation of a Classifier

  • Thus, when the classifier classified a test data set with imbalanced class distributions, then predictive accuracy on its own is not a reliable indicator of a classifier’s effectiveness.

​

  • This necessitates an alternative metrics to judge the classifier.

​

  • Before exploring them, we introduce the concept of Confusion matrix.

​

​

​

BDSET Course

31

DSamanta@IIT Kharagpur

32 of 66

Confusion Matrix

  • A confusion matrix for a two classes (+, -) is shown below.

​

​

​

​

​

  • There are four quadrants in the confusion matrix, which are symbolized as below.
    • True Positive (TP: f++) : The number of instances that were positive (+) and correctly classified as positive (+v).

​

    • False Negative (FN: f+-): The number of instances that were positive (+) and incorrectly classified as negative (-). It is also known as Type 2 Error.

​

    • False Positive (FP: f-+): The number of instances that were negative (-) and incorrectly classified as (+). This also known as Type 1 Error.

​

    • True Negative (TN: f--): The number of instances that were negative (-) and correctly classified as (-).

​

BDSET Course

32

DSamanta@IIT Kharagpur

33 of 66

Confusion Matrix

Note:

  • Np = TP (f++) + FN (f+-)

= is the total number of positive instances.

​

  • Nn = FP(f-+) + Tn(f--)

= is the total number of negative instances.

​

  • N = Np + Nn

= is the total number of instances.

​

  • (TP + TN) denotes the number of correct classification

​

  • (FP + FN) denotes the number of errors in classification.

​

  • For a perfect classifier FP = FN = 0, that is, there would be no Type 1 or Type 2 errors.

​

BDSET Course

33

DSamanta@IIT Kharagpur

34 of 66

Confusion Matrix

Example 11.4: Confusion matrix

​

A classifier is built on a dataset regarding Good and Worst classes of stock markets. The model is then tested with a test set of 10000 unseen instances. The result is shown in the form of a confusion matrix. The result is self explanatory.

​

BDSET Course

34

DSamanta@IIT Kharagpur

Class

Good

Worst

Total

Rate(%)

Good

6954

46

7000

99.34

Worst

412

2588

3000

86.27

Total

7366

2634

10000

95.52

Predictive accuracy?

35 of 66

Confusion Matrix for Multiclass Classifier

  • Having m classes, confusion matrix is a table of size m×m , where, element at (i, j) indicates the number of instances of class i but classified as class j.

​

  • To have good accuracy for a classifier, ideally most diagonal entries should have large values with the rest of entries being close to zero.

​

  • Confusion matrix may have additional rows or columns to provide total or recognition rates per class.

BDSET Course

35

DSamanta@IIT Kharagpur

36 of 66

Confusion Matrix for Multiclass Classifier

​

Example 11.5: Confusion matrix with multiple class

​

Following table shows the confusion matrix of a classification problem with six classes labeled as C1, C2, C3, C4, C5 and C6.

​

BDSET Course

36

DSamanta@IIT Kharagpur

Class

C1

C2

C3

C4

C5

C6

C1

52

10

7

0

0

1

C2

15

50

6

2

1

2

C3

5

6

6

0

0

0

C4

0

2

0

10

0

1

C5

0

1

0

0

7

1

C6

1

3

0

1

0

24

Predictive accuracy?

37 of 66

Confusion Matrix for Multiclass Classifier

  • In case of multiclass classification, sometimes one class is important enough to be regarded as positive with all other classes combined together as negative.

​

  • Thus a large confusion matrix of m*m can be concised into 2*2 matrix.

​

Example 11.6: m×m CM to 2×2 CM

​

  • For example, the CM shown in Example 11.5 is transformed into a CM of size 2×2 considering the class C1 as the positive class and classes C2, C3, C4, C5 and C6 combined together as negative.

​

​

​

​

​

How we can calculate the predictive accuracy of the classifier model in this case?

Are the predictive accuracy same in both Example 11.5 and Example 11.6?

​

​

​

​

​

​

​

​

​

BDSET Course

37

DSamanta@IIT Kharagpur

Class

+

-

+

52

18

-

21

123

38 of 66

Performance Evaluation Metrics

  •  

BDSET Course

38

DSamanta@IIT Kharagpur

39 of 66

Performance Evaluation Metrics

  •  

BDSET Course

39

DSamanta@IIT Kharagpur

40 of 66

Performance Evaluation Metrics

BDSET Course

40

DSamanta@IIT Kharagpur

 

41 of 66

Precision and Recall

BDSET Course

41

DSamanta@IIT Kharagpur

Precision can be seen as a measure of quality.

Recall as a measure of quantity. 

Higher precision means that an algorithm returns more relevant results than irrelevant ones.

Higher recall means that an algorithm returns most of the relevant results (whether or not irrelevant ones are also returned).

42 of 66

Performance Evaluation Metrics

  •  

BDSET Course

42

DSamanta@IIT Kharagpur

43 of 66

Performance Evaluation Metrics

  •  

BDSET Course

43

DSamanta@IIT Kharagpur

Metric

​

​

​

​

Recall

1

1

0

1

Precision

1

0

1

0

​

​

​

1

0

44 of 66

Predictive Accuracy (ε)

  •  

BDSET Course

44

DSamanta@IIT Kharagpur

45 of 66

 

  •  

BDSET Course

45

DSamanta@IIT Kharagpur

46 of 66

Accuracy, Sensitivity and Specificity

  •  

BDSET Course

46

DSamanta@IIT Kharagpur

47 of 66

Sensitivity and Specificity

BDSET Course

47

DSamanta@IIT Kharagpur

Sensitivity refers to a test's ability to designate an individual with disease as positive.

A highly sensitive test means that there are few false negative results, and thus fewer cases of disease are missed.

Specificity refers to a test's ability to designate an individual who does not have a disease as negative.

A high specificity test means that there less miss classification.

​

48 of 66

Analysis with Performance Measurement Metrics

  •  

BDSET Course

48

DSamanta@IIT Kharagpur

 

 

Predicted Class

 

 

+

-

Actual

class

+

P

0

-

0

N

49 of 66

Analysis with Performance Measurement Metrics

  •  

BDSET Course

49

DSamanta@IIT Kharagpur

 

 

Predicted Class

 

 

+

-

Actual

class

+

0

P

-

N

0

50 of 66

Analysis with Performance Measurement Metrics

  •  

BDSET Course

50

DSamanta@IIT Kharagpur

 

 

Predicted Class

 

 

+

-

Actual

class

+

P

0

-

N

0

51 of 66

Analysis with Performance Measurement Metrics

  •  

BDSET Course

51

DSamanta@IIT Kharagpur

 

 

Predicted Class

 

 

+

-

Actual

class

+

0

p

-

0

N

52 of 66

Predictive Accuracy versus TPR and FPR

  • One strength of characterizing a classifier by its TPR and FPR is that they do not depend on the relative size of P and N.

​

    • The same is also applicable for FNR and TNR and others measures from CM.

​

  • In contrast, the Predictive Accuracy, Precision, Error Rate, F1 Score, etc. are affected by the relative size of P and N.

​

  • FPR, TPR, FNR and TNR are calculated from the different rows of the CM.

​

    • On the other hand Predictive Accuracy, etc. are derived from the values in both rows.

​

  • This suggests that FPR, TPR, FNR and TNR are more effective than Predictive Accuracy, etc.

BDSET Course

52

DSamanta@IIT Kharagpur

53 of 66

BDSET Course

53

DSamanta@IIT Kharagpur

ROC Curves

54 of 66

ROC Curves

  • ROC is an abbreviation of Receiver Operating Characteristic come from the signal detection theory, developed during World War 2 for analysis of radar images.

​

  • In the context of classifier, ROC plot is a useful tool to study the behaviour of a classifier or comparing two or more classifiers.

​

  • A ROC plot is a two-dimensional graph, where, X-axis represents FP rate (FPR) and Y-axis represents TP rate (TPR).

​

  • Since, the values of FPR and TPR varies from 0 to 1 both inclusive, the two axes thus from 0 to 1 only.

​

  • Each point (x, y) on the plot indicating that the FPR has value x and the TPR value y.

​

​

BDSET Course

54

DSamanta@IIT Kharagpur

55 of 66

ROC Plot

  • A typical look of ROC plot with few points in it is shown in the following figure.

​

​

​

​

​

​

​

​

  • Note the four cornered points are the four extreme cases of classifiers

​

Identify the four extreme classifiers.

BDSET Course

55

DSamanta@IIT Kharagpur

56 of 66

Interpretation of Different Points in ROC Plot

  • Le us interpret the different points in the ROC plot.

​

​

​

​

​

​

​

​

​

​

  • The four points (A, B, C, and D)

​

    • A: TPR = 1, FPR = 0, the ideal model, i.e., the perfect classifier, no false results

​

    • B: TPR = 0, FPR = 1, the worst classifier, not able to predict a single instance

​

    • C: TPR = 0, FPR = 0, the model predicts every instance to be a Negative class, i.e., it is an ultra-conservative classifier

​

    • D: TPR = 1, FPR = 1, the model predicts every instance to be a Positive class, i.e., it is an ultra-liberal classifier

​

​

​

BDSET Course

56

DSamanta@IIT Kharagpur

57 of 66

Interpretation of Different Points in ROC Plot

  • Le us interpret the different points in the ROC plot.

​

​

​

​

​

​

​

​

​

​

  • The points on diagonals

​

    • The diagonal line joining point C(0,0) and D(1,1) corresponds to random guessing

​

      • Random guessing means that a record is classified as positive (or negative) with a certain probability
      • Suppose, a test set contacting N+ positive and N- negative instances. Suppose, the classifier guesses any instances with probability p
      • Thus, the random classifier is expected to correctly classify p.N+ of the positive instances and p.N- of the negative instances
      • Hence, TPR = FPR = p
      • Since TPR = FPR, the random classifier results reside on the main diagonals

​

​

​

​

BDSET Course

57

DSamanta@IIT Kharagpur

58 of 66

Interpretation of Different Points in ROC Plot

  • Let us interpret the different points in the ROC plot.

​

​

​

​

​

​

​

​

​

  • The points on the upper diagonal region

​

    • All points, which reside on upper-diagonal region are corresponding to classifiers “good” as their TPR is as good as FPR (i.e., FPRs are lower than TPRs)

​

    • Here, X is better than Z as X has higher TPR and lower FPR than Z.

​

    • If we compare X and Y, neither classifier is superior to the other

​

​

​

​

​

BDSET Course

58

DSamanta@IIT Kharagpur

59 of 66

Interpretation of Different Points in ROC Plot

  • Let us interpret the different points in the ROC plot.

​

​

​

​

​

​

​

​

​

  • The points on the lower diagonal region

​

    • The Lower-diagonal triangle corresponds to the classifiers that are worst than random classifiers

​

    • Note: A classifier that is worst than random guessing, simply by reversing its prediction, we can get good results.

​

      • W’(0.2, 0.4) is the better version than W(0.4, 0.2), W’ is a mirror reflection of W

​

​

​

​

​

BDSET Course

59

DSamanta@IIT Kharagpur

60 of 66

Tuning a Classifier through ROC Plot

  • Using ROC plot, we can compare two or more classifiers by their TPR and FPR values and this plot also depicts the trade-off between TPR and FPR of a classifier.

​

​

​

​

​

​

​

  • Examining ROC curves can give insights into the best way of tuning parameters of classifier.

​

  • For example, in the curve C2, the result is degraded after the point P. Similarly for the observation C1, beyond Q the settings are not acceptable.

BDSET Course

60

DSamanta@IIT Kharagpur

61 of 66

Comparing Classifiers trough ROC Plot

  • Two curves C1 and C2 are corresponding to the experiments to choose two classifiers with their parameters.

​

  • Here, C2 is better than C1 when FPR is less than 0.3.

​

  • However, C1 is better, when FPR is greater than 0.3.

​

  • Clearly, neither of these two classifiers dominates the other.

BDSET Course

61

DSamanta@IIT Kharagpur

62 of 66

Comparing Classifiers trough ROC Plot

  • We can use the concept of “area under curve” (AUC) as a better method to compare two or more classifiers.

​

  • If a model is perfect, then its AUC = 1.

​

  • If a model simply performs random guessing, then its AUC = 0.5

​

  • A model that is strictly better than other, would have a larger value of AUC than the other.

​

​

​

​

​

​

​

  • Here, C3 is best, and C2 is better than C1 as AUC(C3)>AUC(C2)>AUC(C1).

​

​

​

​

BDSET Course

62

DSamanta@IIT Kharagpur

63 of 66

A Quantitative Measure of a Classifier

  •  

BDSET Course

63

DSamanta@IIT Kharagpur

64 of 66

A Quantitative Measure of a Classifier

  •  

BDSET Course

64

DSamanta@IIT Kharagpur

65 of 66

Reference

BDSET Course

65

DSamanta@IIT Kharagpur

  • The detail material related to this lecture can be found in

​

Data Mining: Concepts and Techniques, (3rd Edn.), Jiawei Han, Micheline Kamber, Morgan Kaufmann, 2015.

​

Introduction to Data Mining, Pang-Ning Tan, Michael Steinbach, and Vipin Kumar, Addison-Wesley, 2014

66 of 66

Any question?

​

​

BDSET Course

66

DSamanta@IIT Kharagpur