1 of 46

Skin Lesion Classification Explained with�Generative Adversarial Networks

1

Candidate:

Ettore Candeloro

Supervisor:

Prof. Costantino Grana

Co-Supervisors:

Prof. Federico Bolelli

Dott. Luca Lumetti

University of Modena and Reggio Emilia

Dep. of Engineering “Enzo Ferrari”

A.A. 2022/2023

2 of 46

1/32

3 of 46

2/32

3 Mln affected people

​

Over 59,000 deaths/year

​

Diverse risk factors

Melanoma

4 of 46

3/32

99% - localized cancer

​

35% - non-localized cancer

Melanoma

5 of 46

Identikit of a Melanoma

6 of 46

5/32

ABCD Features

Melanoma

Nevus

7 of 46

6/32

ABCD Features

Asymmetry

8 of 46

7/32

ABCD Features

Borders

9 of 46

8/32

ABCD Features

Colors

10 of 46

9/32

ABCD Features

Diameter

11 of 46

10/32

Dermoscopy

12 of 46

Computer Aided Diagnosis �for Skin Lesions

13 of 46

12/32

CAD AI Systems

GOAL: Assisted skin lesion diagnosis

IDEA: use CNNs

14 of 46

13/32

Skin lesion classification

Melanoma!

CNN

15 of 46

14/32

Nevus!

CNN

Skin lesion classification

16 of 46

15/32

Nevus!

CNN

BUT WHY?

Skin lesion classification

17 of 46

16/32

Problems

Black-box nature of DL algorithms used

No explanations about the diagnosis

18 of 46

Explainability for �CNN Classifiers

19 of 46

18/32

Heat-Map Based Methods

20 of 46

19/32

Heat-Map Based Methods

What visual feature? Which aspect of it?

21 of 46

20/32

Counterfactual visual explanations with GANs

If input X were X’ output Y would change to Y’

Class → LION

Class → CHEETAH

22 of 46

21/32

Counterfactual visual explanations with GANs

Class → LION

Class → CHEETAH

What single visual attribute?

Which contributed the most?

23 of 46

22/32

StylEx

Top-N classifier specific attributes

24 of 46

23/32

StylEx architecture & training

25 of 46

24/32

Generating Counterfactuals with StylEx

26 of 46

StylEx for Skin Lesion Classifier’s Explainability

27 of 46

26/32

Setup

ISIC 2019 + 2020 Dataset

ResNet50 CNN

Binary classification

Re-StylEx unofficial code-base

28 of 46

27/32

CNN Training

Data Augmentation

Weighted Random Sampling

Low Resolution of

128x128 pixels

MEL Recall = 0.800

F1 Avg = 0.867

29 of 46

28/32

StylEx Training

Frechét Inception Distance (FID)

30 of 46

29/32

Training Results: interpolation in the StyleSpace

31 of 46

30/32

Counterfactuals results

COLOR HUE

COLOR UNIFORMITY

PINK COLOR

BORDERS/SCARRING

1.00

0.02

1.00

1.00

0.258

0.973

0.88

0.00

32 of 46

31/32

Comparison with Re-Stylex

33 of 46

32/32

Considerations & Future Work

Some visual features aligned with ABCD

Others were uncorrelated with melanoma features

Using SOTA skin lesion classifiers, with high res. images

User tests & collaboration with trained dermatologists

34 of 46

Thanks for your attention

35 of 46

Backup Slides

36 of 46

3 risk levels with different requisites:

low, high and prohibited

Medical CAD AI applications are high-risk

The EU AI Act

EU proposed AI regulatory framework

37 of 46

High Risk AI Requirements

Transparency

Robustness

Accuracy

Data quality

Provide info

Human oversight

38 of 46

38

Metrics

39 of 46

StyleGAN

40 of 46

Counterfactual Explanations with GANs

SUV

SEDAN

If input X were X’ , output Y would change to Y’

41 of 46

StylEx classifier related attributes example

42 of 46

StylEx Architecture detailed

43 of 46

StylEx Losses in details

44 of 46

StylEx Rec scaling factor effects

=1

=10

45 of 46

Full interpolation in the StyleSpace

46 of 46

Counterfactuals: top class attributes