1 of 58

eXplainable AI (XAI)

2 of 58

Black Box AI

AI produces insights based on a data set, but the end-user doesn’t know how

AI does not provide reasons behind the decision or prediction it makes

2

Input

Output

“Dog”

3 of 58

Black Box AI

AI produces insights based on a data set, but the end-user doesn’t know how

AI does not provide reasons behind the decision or prediction it makes

3

Input

Black Box

Output

“Dog”

4 of 58

Feature Importance

Feature importance provides insights into how individual features impact the model’s output

4

Input

Machine Learning

Output

5 of 58

Feature Importance

Feature importance provides insights into how individual features impact the model’s output

5

Input

Machine Learning

Output

Feature importance

6 of 58

Feature Importance

In regression, feature importance reflects how much each predictor (input variable) contributes to the model's output

Process engineer

To understand not only how the current production settings affect product quality but also which control parameters need adjustment when production targets are not met

6

7 of 58

Feature Importance

In decision tree

Process engineer

To understand not only how the current production settings affect product quality but also which control parameters need adjustment when production targets are not met

7

Feature 1

Good: 0

Bad: 3

D = 0

Good: 1

Bad: 0

D = 0

0

1

2

Feature 2

Good: 0

Bad: 2

D = 0

Good: 2

Bad: 0

D = 0

0

1

Feature 1	Feature 2	Feature 3	Feature 4	output
?	Yes	Low	Medium	Bad
Yes	Yes	High	Medium	Bad
?	No	High	Medium	Good
No	No	Medium	High	Good
?	No	Medium	Low	Good
Yes	No	Low	High	Bad
Yes	No	Medium	High	Bad
?	Yes	High	Low	Bad

8 of 58

eXplainable AI

A set of methods and techniques that make the decision-making processes of AI systems

transparent,
interpretable, and
understandable to humans

Depending on the AI performance, XAI results can be used in various ways:

If AI performance < Human performance ⟹ XAI suggests improvement directions for AI models
If AI performance ≈ Human performance ⟹ XAI identifies the principles behind AI model learning
If AI performance > Human performance ⟹ XAI enables acquiring new knowledge from AI

8

9 of 58

Methods for Explainable AI

Post-Hoc Explanation Methods: Techniques that explain decisions after the model has made predictions

SHAP
LIME
Grad-CAM

Intrinsically Interpretable Models: Models that are inherently interpretable due to their design

Linear Regression and Decision Trees
CAM

9

10 of 58

SHapley Additive exPlanations (SHAP)

10

11 of 58

SHapley Additive exPlanations (SHAP)

SHAP: a game theoretic approach to explain the output of any machine learning model

Compute feature importance on each predicted value using Shapley value
According to the SHAP value, the contribution of each feature can be expressed as the degree of change in the overall performance when the contribution of that feature is excluded

11

Local Feature Importance

Global Feature Importance

12 of 58

Feature Importance

We will not provide an in-depth discussion of SHAP, as it is already comprehensively implemented in widely available Python libraries

12

Input

Machine Learning

Output

SHAP

Feature importance

13 of 58

Lab: Wind Turbine Energy Prediction

Output

LV ActivePower (kW): The actual electrical power generated by the turbine at that moment

Inputs

Wind Speed (m/s)
Wind Direction (°)
Month
Hour
Season
DayNight
Temperature
Theoretical_Power_Curve (kW): The theoretical power output, based on the turbine manufacturer’s specifications for a given wind speed.

13

14 of 58

Lab: Wind Turbine Energy Prediction

Output

LV ActivePower (kW): The actual electrical power generated by the turbine at that moment

Inputs

Wind Speed (m/s)
Wind Direction (°)
Month
Hour
Season
DayNight
Temperature
Theoretical_Power_Curve (kW): The theoretical power output, based on the turbine manufacturer’s specifications for a given wind speed.

14

15 of 58

Python: Wind Turbine Energy Prediction

Train a model first

Prediction

15

16 of 58

Python: SHAP

Global Feature Importance
While SHAP can explain individual predictions (local interpretability), it can be used to measure global feature importance by averaging the absolute SHAP values across the entire test dataset

16

17 of 58

Local Interpretable Model-agnostic Explanations �(LIME)

17

18 of 58

Local Interpretable Model-agnostic Explanations (LIME)

A technique used to explain individual predictions made by any machine learning model by approximating the complex model locally with a simpler, interpretable model (e.g., a linear regression model or a decision tree)

Key idea: like sensitivity analysis

Perturbing the input features around a specific instance and observing how the model’s output changes

18

19 of 58

Local Interpretable Model-agnostic Explanations (LIME)

Explain with image

LIME partially masks the input data
Then, the transformed input data is given to the model and the predicted value is obtained
If the prediction result changed a lot, we can know that the masked part was important
On the other hand, if the prediction result did not change much, we can know that the masked part was not very important

19

Marco Tulio Ribeiro, S. S. (2016, August 12). Local interpretable model-agnostic explanations (LIME): An introduction. O'Reilly Media.

20 of 58

How LIME Works

Perturbation:

LIME creates slightly modified versions of the input (perturbed samples) by randomly changing the feature values
These perturbed samples are then passed through the model to get predictions

Simple Model Fitting:

A simple interpretable model (like linear regression) is fit to approximate the relationship between the features and the model’s output in the local neighborhood of the instance

Feature Importance:

The coefficients of the simple model indicate the importance of features in making the prediction for the given instance

20

21 of 58

LIME

21

22 of 58

Saliency Maps in Explainable AI

23 of 58

Saliency Maps

XAI methods designed specifically for image-based models

When we visually identify an object in an image

We do not examine the entire image
Rather, we intuitively focus on the most important parts

Saliency maps highlight the most “salient” (important) pixels or regions of an input image that the model relies on to make its prediction

Saliency maps: Visualizing convolutional neural networks

23

24 of 58

What Regions is AI Focusing On?

24

Activation map (like a heat map)

25 of 58

What Regions is AI Focusing On?

25

26 of 58

Grad-CAM: �Gradient-weighted Class Activation Maps

26

27 of 58

Saliency Maps in Explainable AI

When we visually identify an object in an image, we do not examine the entire image; rather,

We intuitively focus on the most important parts.
Similarly, CNN learning mimics this human behavior by assigning higher weights to the most relevant parts during optimization.

CNNs typically cannot explicitly recognize these important regions
Saliency maps play a crucial role by visually illustrating which regions of an image are most influential in the model's decision-making process.

27

28 of 58

Key Concept of Saliency Maps

28

29 of 58

Grad-CAM: Gradient-weighted Class Activation Maps

A powerful visualization tool in explainable AI used to understand the decisions of convolutional neural networks (CNNs).
Grad-CAM produces heatmaps that highlight the regions in the input image that were most relevant for the model's prediction of a specific class
Grad-CAM uses the gradients of the class score with respect to the feature maps of a convolutional layer to identify the important regions of the image

29

Selvaraju et al. Grad-CAM: Visual explanations from deep networks

via gradient-based localization. ICCV’17

30 of 58

Grad-CAM: Gradient-weighted Class Activation Maps

30

31 of 58

Step 1: Train CNN

Image Classification Problem

31

Convolution

Max pooling

…

Flatten

Convolution and pooling layers

Convolutional Neural Networks

Classification

Fully connected layer

7

9

…

Last Feature Maps

32 of 58

Step 2: Forward Pass through the trained CNN

32

2

3

7

0

-7

-3

-2

2

-2

9

1

5

0

-2

6

-1

4

-2

4

-5

7

8

0

3

6

4

2

-2

2

-9

5

…

Flatten

1

2

4

1

6

7

8

5

2

0

3

2

3

4

1

…

softmax

Fully connected

Last Feature Maps

33 of 58

Step 3: Compute Gradients and Global Average of the Gradients

33

2

3

7

0

-7

-3

-2

2

-2

9

1

5

0

-2

6

-1

4

-2

4

-5

7

8

0

3

6

4

2

-2

2

-9

5

…

Flatten

1

2

4

1

6

7

8

5

2

0

3

2

3

4

1

softmax

Last Feature Maps

Fully connected

34 of 58

Step 4: Weight the Feature Maps

34

2

3

7

0

-7

-3

-2

2

-2

9

1

5

0

-2

6

-1

4

-2

4

-5

7

8

0

3

6

4

2

-2

2

-9

5

…

Flatten

1

2

4

1

6

7

8

5

2

0

3

2

3

4

1

Class Activation Map

softmax

Last Feature Maps

Fully connected

35 of 58

Lab for Grad-CAM: MNIST

Image Classification Problem

35

Convolution

Max pooling

…

Flatten

Convolution and pooling layers

Convolutional Neural Networks

Classification

Fully connected layer

7

9

…

Feature Maps

36 of 58

Lab for Grad-CAM: MNIST

Image Classification Problem

36

Convolution

Max pooling

…

Flatten

Convolution and pooling layers

Convolutional Neural Networks

Classification

Fully connected layer

7

9

…

Feature Maps

Class Activation Map

37 of 58

Python: Grad-CAM

Train CNN

37

38 of 58

Python: Grad-CAM

For test image, compute saliency map

38

39 of 58

Result: Grad-CAM

Resize and overlap

39

Original test image

Saliency map

Resized

Overlapped

40 of 58

CAM: �Class Activation Maps

40

41 of 58

Visualizing Convolutional Neural Networks

Class Activation Map (CAM) for a given class highlights the image regions used by the CNN to identify that class

41

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba.

Learning Deep Features for Discriminative Localization. CVPR'16

42 of 58

Fully Connected Layer

Image Classification Problem

42

Convolution

Max pooling

…

Flatten

Convolution and pooling layers

Convolutional Neural Networks

Classification

Fully connected layer

7

9

…

Last Feature Maps

43 of 58

Global Average Pooling (GAP)

Unlike fully connected (dense) layers, which flatten the feature maps, GAP condenses each feature map into a single value by taking the spatial average

43

1

2

4

1

6

7

8

5

2

1

0

3

2

3

4

1

Global Avg pooling

3.125

44 of 58

Global Average Pooling (GAP)

Unlike fully connected (dense) layers, which flatten the feature maps, GAP condenses each feature map into a single value by taking the spatial average

44

2

3

7

0

-7

-3

-2

2

-2

9

1

5

0

-2

6

-1

4

-2

4

-5

7

8

0

3

6

4

2

-2

2

-9

5

1

2

4

1

6

7

8

5

2

1

0

3

2

3

4

1

3.125

1.625

1.25

45 of 58

Train CNN with Global Average Pooling

Image Classification Problem

45

Convolution

Max pooling

…

7

9

GAP

Convolution and pooling layers

Convolutional Neural Networks

Classification

GAP

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba,

Learning Deep Features for Discriminative Localization, CVPR'16

46 of 58

Feed New Image into the Trained Network

Once training is complete,

46

Convolution

Max pooling

…

7

9

GAP

Convolution and pooling layers

Convolutional Neural Networks

Classification

GAP

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba,

Learning Deep Features for Discriminative Localization, CVPR'16

47 of 58

Feed New Image into the Trained Network

Once training is complete, a new input image is fed into the trained network

47

Convolution

Max pooling

…

7

9

GAP

Convolution and pooling layers

Convolutional Neural Networks

Classification

GAP

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba,

Learning Deep Features for Discriminative Localization, CVPR'16

48 of 58

Feed New Image into the Trained Network

Once training is complete, a new input image is fed into the trained network

48

Convolution and pooling layers

Convolutional Neural Networks

Classification

GAP

Convolution

Max pooling

…

7

9

GAP

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba,

Learning Deep Features for Discriminative Localization, CVPR'16

49 of 58

Class Activation Map (CAM)

Weighted Sum of Feature Maps

49

Convolution and pooling layers

Convolutional Neural Networks

Classification

GAP

Convolution

Max pooling

…

7

9

GAP

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba,

Learning Deep Features for Discriminative Localization, CVPR'16

Class Activation Map

50 of 58

Resize and Overlay

Resize CAM to match the original image
Overlay CAM on the original image

50

Convolution and pooling layers

Convolutional Neural Networks

Classification

GAP

Convolution

Max pooling

…

7

9

GAP

Resize and overlay

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba,

Learning Deep Features for Discriminative Localization, CVPR'16

Class Activation Map

51 of 58

Implementation: Global Average Pooling

51

52 of 58

Implementation: CAM Heatmap

52

1) Predict the class label

2) Select the corresponding CAM based on the predicted class

53 of 58

CAM Heatmap

53

Resize

Overlay

54 of 58

Steel Surface Defects

54

55 of 58

Steel Surface Defects

55

Patches

Inclusion

Scratches

Steel Surface Defect Diagnostics using Deep Convolutional Neural Network and Class Activation Map, Applied Sciences

56 of 58

Steel Surface Defects

56

Patches

Inclusion

Scratches

Steel Surface Defect Diagnostics using Deep Convolutional Neural Network and Class Activation Map, Applied Sciences

Defect regimes are never labeled

57 of 58

Cantilever

57

In collaboration with KIMM

58 of 58

Visually Identify the Region with the Highest Vibration

58

In collaboration with KIMM