1 of 82

Trustworthy ML

Winter Semester 2022-2023

University of Tübingen

Lecturer : Seong Joon Oh

2 of 82

Reminder for video recording

3 of 82

Exercise 1 grades under way

4 of 82

No tutorial after the lecture

But feel free to ask questions.

5 of 82

Attribution to test features

Spatial pooling

Input image

Score map

Class label

CNN

Model

Cat

GAP

Thresholding

FG-BG mask

Zhou et al. Learning deep features for discriminative localization. CVPR’16.

6 of 82

Attribution to test features

Spatial pooling

Input image

Score map

Class label

CNN

Model

Cat

GAP

Thresholding

FG-BG mask

Zhou et al. Learning deep features for discriminative localization. CVPR’16.

What is it?

7 of 82

Attribution to test features

Spatial pooling

Input image

Score map

Class label

CNN

Model

Cat

GAP

Thresholding

FG-BG mask

Zhou et al. Learning deep features for discriminative localization. CVPR’16.

Why is it a cat?

8 of 82

Which features?

For visual models taking image as input:

Single pixel

Image patch

Superpixel

Instance mask

9 of 82

Which features?

For visual models taking image as input:

Cute 90%

Furry 40%

Yellow eyes 50%

Animal 100%

Two ears 100%

Pet 70%

Attributes

Why a cat?

10 of 82

Which features?

For language models taking a token sequence as inputs:

Individual tokens / words are popular candidates for explanation unit.

A Song of (Dis) agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing. 2022.

11 of 82

Counterfactual reasoning over features

Input

Is this still predicted as a cat if this feature is missing?

Cat?

12 of 82

Counterfactual reasoning over features

Input

Is this still predicted as a cat if this feature is missing?

What do we mean by “missing” ?

Black pixels? Gray pixels? Pink pixels?

Inpainting based on the context?

Cat?

13 of 82

Counterfactual reasoning over features

Input

Is this still predicted as a cat if this feature is replaced with something else?

Cat?

14 of 82

Counterfactual reasoning over features

Output

Which features contribute to predicting a cat rather than a dog?

Cat? Dog?

15 of 82

Intrinsically interpretable models support this.

We know the full effect of changing one or many features on the output.
Implicit aim: linearise our complex model in some way for interpretability.

16 of 82

For neural networks: expensive computation.

For each image, one needs to compute N forward operations through an NN.
N = # Sliding windows per image X # Ways to alter the window content

GSWO: A Programming Model for GPU-enabled Parallelization of Sliding Window Operations in Image Processing. 2016

17 of 82

Special case. Feature: pixel. Perturbation: small.

Original pixel value: (232,216,231)

New pixel value: (233,216,231)

Original cat score: 96.5%

New cat score: 96.4%

18 of 82

Special case. Feature: pixel. Perturbation: small.

Dumb way: Compute forward pass #pixels X 2 times to measure pixel-wise infinitesimal contribution.
Smart way: Compute one backward pass (efficient thanks to backprop).

Original pixel value: (232,216,231)

New pixel value: (233,216,231)

Original cat score: 96.5%

New cat score: 96.4%

19 of 82

Special case. Feature: pixel. Perturbation: small.

Backprop linearises the whole model around the test sample.
Local explanation with linear surrogate model.
One can freely compute input- and output-based counterfactuals.

20 of 82

Special case. Feature: pixel. Perturbation: small.

SmoothGrad: removing noise by adding noise. 2017.

21 of 82

Issue with input gradients

Gradient maps are often too noisy.
Can we get smoother maps?

SmoothGrad: removing noise by adding noise. 2017.

Input

Input grad

22 of 82

Smoothgrad: Smoother input gradients.

Compute gradients in the vicinity of the input x: x + ε.
Average them.
Slightly less local than vanilla gradient.

SmoothGrad: removing noise by adding noise. 2017.

Input

Smoothgrad

Input grad

23 of 82

Local gradients

f(x)

x+δe_i

f(x+δe_i)

f(x+δe_i) - f(x) ≈ < δe_i, df/dx > = δ x df/dx_i

Vanilla grad: Measure contribution of pixel i with infinitesimal counterfactual.

Changes in infinitesimally local area

24 of 82

Less local gradients

x+z₂

x+z₂+δe_i

Smoothgrad: Measure contribution of pixel i with infinitesimal (δ) counterfactual at multiple points (x+z) around x.

f(x+z₂)

f(x+z₂+δe_i)

x+z₁

x+z₃

x+z₄

Changes in broader

local area

25 of 82

Towards global gradients: Integrated Gradients

f(x+α (x’ - x))

Integrated gradients

x + α (x’ - x)

x’

Changes in global area

f(x+α (x’ - x)+δe_i)

26 of 82

Towards global gradients: Integrated Gradients

Formula we derived for contribution of pixel i.

Contribution of pixel i computed by “Integrated Gradients”.

Axiomatic Attribution for Deep Networks. ICML 2017.

Why do we wish to compute “Integrated Gradients” ?

27 of 82

Towards global gradients: Integrated Gradients

Axiomatic Attribution for Deep Networks. ICML 2017.

… and then if we sum over all pixels i, we have

When we define the contribution from pixel i as follows …

28 of 82

Towards global gradients: Integrated Gradients

We note the following: the fundamental theorem of line integrals.

Axiomatic Attribution for Deep Networks. ICML 2017.

Let’s plug below in.

Then, we derive the following property:

29 of 82

Towards global gradients: Integrated Gradients

Completeness Axiom (“necessary condition” for a sound attribution):

Pixel-wise contributions for x must sum up to the difference between the current model output f(x) and some baseline output f(x⁰).
Baseline: Image without any information.

Integrated gradients satisfy the “Completeness Axiom”.

Axiomatic Attribution for Deep Networks. ICML 2017.

30 of 82

Towards global gradients: Integrated Gradients

How to set the baseline x⁰?
Usually use “black image”: x⁰=0.
Downside: black pixels (e.g. pixels for black camera) cannot be attributed at all because x_i-x_i⁰ = 0.
Is this right?

Axiomatic Attribution for Deep Networks. ICML 2017.

31 of 82

Two ways of measuring contribution

Local perturbation

Response to local perturbations

Response to global perturbations

“Turning off” feature 2

x₁

x₂

Input gradient

SmoothGrad

Integrated gradients

LIME

SHAP

Zintgraf et al.

32 of 82

Local == Global for sparse linear models.

Local perturbation

Response to local perturbations

Response to global perturbations

“Turning off” feature 2

x₁

x₂

Gradient of output wrt feature i is c_i .

Effect of “turning off” feature i is c_iφ_i.

33 of 82

Zintgraf et al: Inpaint + Black box computation

Visualizing deep neural network decisions: Prediction difference analysis. ICLR 2017.

Classification score after removing feature i.

And then compute the counterfactual before and after removing feature i.

Approximate this using an inpainting model

34 of 82

Recall: This is expensive.

For each image, one needs to compute N forward operations for the main model + N inpainting computations.
N = #Sliding windows X #Samples for inpainting

GSWO: A Programming Model for GPU-enabled Parallelization of Sliding Window Operations in Image Processing. 2016

35 of 82

Zintgraf et al: Inpaint + Black box computation

Visualizing deep neural network decisions: Prediction difference analysis. ICLR 2017.

36 of 82

Zintgraf et al: Inpaint + Black box computation

Left: input grad.

Right: Zintgraf et al.

Pro:

Global counterfactual.

Con:

Expensive.
Depends on inpainter.

Visualizing deep neural network decisions: Prediction difference analysis. ICLR 2017.

37 of 82

LIME: Fit a sparse linear model.

f: original model
g: surrogate model
π_x: measure of distance from x � g needs to be close to f only in the vicinity of x.
Ω: measure of complexity

"Why should i trust you?" Explaining the predictions of any classifier. KDD 2016.

38 of 82

LIME: Fit a sparse linear model.

Given the general formulation

We specify sparse linear function g by defining:

linearity

sparsity constraint

The function fitting takes place around input x:

Let g follow f (L2 loss) ...

… in the vicinity of x.

"Why should i trust you?" Explaining the predictions of any classifier. KDD 2016.

39 of 82

LIME: Fit a sparse linear model.

"Why should i trust you?" Explaining the predictions of any classifier. KDD 2016.

40 of 82

LIME: Fit a sparse linear model.

Pro: Results are interpretable.

Con: Approximation. Expensive. � Reference is assumed to be a gray image.

"Why should i trust you?" Explaining the predictions of any classifier. KDD 2016.

41 of 82

Shapley values

Assume we have a black box system with binary input vector.

How do you assign the contribution of each feature i ?

System

Input

[0 1 0 0 1 0]

Output

5.27

42 of 82

Shapley values

Binary input features have a clear interpretation of turning on (1) and turning off (0).

The Shapley value determines the individual contribution of each feature. The value is defined as:

Here, the original input x is always treated as [ 1 1 1 1 1 1 ] and an example of a valid sample z is [ 0 1 0 0 1 0 ] for index i=2 (index starts from 1).

43 of 82

Shapley values

The expectation follows:

Sample subset size m from Unif{1, … , |x|}.
Then, sample a subset z of size m containing feature i with equal probabilities.

44 of 82

Shapley values in deterministic form

The expectation follows:

Sample subset size m from Unif{1, … , |x|}.
Then, sample a subset z of size m with equal probabilities.

45 of 82

Intuitive description of Shapley values

Average function output with for the inputs

with the feature

Average function output with for the inputs

without the feature

MINUS

46 of 82

Nice property of Shapley values

They satisfy the “Completeness Axiom” as well.

Remember Integrated gradients?

47 of 82

Another nice property of Shapley values

Attribution values φ have the strong monotonicity property if, for every function f and f’ and input x and feature i, the following holds

Implies

That is, if the overall impact of deleting i is greater for f’, then the attribution value for f’ on feature i must be greater than that for f.

48 of 82

Shapley values are special

Theorem:

The only way for the attribution values φ to satisfy both the strong monotonicity and the completeness axiom is to take the Shapley values.

Monotonic Solutions of Cooperative Games.

49 of 82

Defining a “missing feature”.

We needed a good definition of “no information” for the following methods:

Integrated gradients
Zintgraf et al.
LIME
Shapley values

What are the typical values taken?

50 of 82

Defining a “missing feature”.

Integrated gradients: Zero (black pixels) as missing features.

Zintgraf et al: Imputation (inpainting) as missing features.

LIME: Mean values (gray pixels) as missing features.

Shapley: Mean values (gray pixels) as missing features.

51 of 82

Problem with fixed missing feature values.

Black and gray pixels also carry information!

52 of 82

Meaningful perturbations

One can also erase information using image blurring.

Interpretable Explanations of Black Boxes by Meaningful Perturbation. ICCV 2017.

53 of 82

Meaningful perturbations

m*: Binary mask that minimally erases information in the image.

||1-m||₁ measures the area of the erased area.

f_c: Classifier output for class c.

Φ(x₀; m): Blurring of image x₀ according to the mask m.

Interpretable Explanations of Black Boxes by Meaningful Perturbation. ICCV 2017.

54 of 82

Meaningful perturbations

Score changes after erasing information with blurring.

→ Lets us know when an algorithm may have learned the wrong association.

Interpretable Explanations of Black Boxes by Meaningful Perturbation. ICCV 2017.

55 of 82

Two ways of measuring contribution

Pro

Well-understood properties.
No dependence on reference values.

Con

Only infinitesimal counterfactuals.

Pro

Meaningful counterfactual analysis.

Con

How to set the reference values?
Computationally heavy; need further assumptions to make it efficient.

Local perturbation

Response to local perturbations

Response to global perturbations

“Turning off” feature 2

x₁

x₂

56 of 82

Revisiting the choice of features

For visual models taking image as input:

Single pixel

Image patch

Superpixel

Instance mask

57 of 82

Methods using different features

LIME uses superpixels.

What are superpixels?

→ Grouping of pixels respecting colour/edge

similarity. Finer than semantic segmentation

but coarser than raw pixels.

"Why should i trust you?" Explaining the predictions of any classifier. KDD 2016.

58 of 82

We probably care about “interpretable” features.