1 of 152

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH)

Alessandro Bombini, INFN-CHNet Firenze

INFN, Pisa, December 6th, 2022

2 of 152

2

AIRES-CH

DNN model

the CHNet AIRES-CH project - Alessandro Bombini

A cloud-native application for digital restoration of Cultural Heritage using nuclear imaging:

the AIRES-CH project App

THESPIAN-XRF app

AIRES-CH paper 1

AIRES-CH paper 2

3 of 152

3

the CHNet AIRES-CH project - Alessandro Bombini

Physics

AI, ML & DNN

Cultural Heritage

[1]

[2],

AIRES-CH

Nice reviews:

​

[1] Giuntini L, Taccetti F, et al.. Detectors and Cultural Heritage: The INFN-CHNet Experience. Applied Sciences. 2021; 11(8):3462. https://doi.org/10.3390/app11083462

[2] M. Feickert, B. Nachman, A Living Review of Machine Learning for Particle Physics, https://github.com/iml-wg/HEPML-LivingReview

​

[3] M. Fiorucci, M. Khoroshiltseva, M. Pontil, A. Traviglia, A. Del Bue, S. James, Machine Learning for Cultural Heritage: A Survey,

Pattern Recognition Letters, Volume 133, 2020, https://doi.org/10.1016/j.patrec.2020.02.017

Nuclear Imaging

Raw Data

of

Pictorial Artworks

Colour

Digital restoration

4 of 152

4

4

the CHNet AIRES-CH project - Alessandro Bombini

STRATIGRAPHY OF PICTORIAL ARTWORKS

support

glue (+ chalk) ground

white priming

preparatory drawing

wood, canvas, etc.

organic glues, etc.

Pb white, Ca white, etc.

graphite

painted layer 1

multi-elemental pigments

painted layer n

multi-elemental pigments

varnish

organic

5 of 152

5

5

VIS/NIR

the CHNet AIRES-CH project - Alessandro Bombini

STRATIGRAPHY OF PICTORIAL ARTWORKS

Physical Imaging techniques

support

glue (+ chalk) ground

X, γ

white priming

preparatory drawing

wood, canvas, etc.

IR

organic glues, etc.

Pb white, Ca white, etc.

graphite

painted layer 1

multi-elemental pigments

painted layer n

multi-elemental pigments

varnish

organic

6 of 152

6

the CHNet AIRES-CH project - Alessandro Bombini

X, γ

STRATIGRAPHY OF PICTORIAL ARTWORKS

Signals

7 of 152

7

the CHNet AIRES-CH project - Alessandro Bombini

Portable XRF

Taccetti, F., Castelli, L., Czelusniak, C. et al. A multipurpose X-ray fluorescence scanner developed for in situ analysis. Rend. Fis. Acc. Lincei 30, 307–322 (2019). https://doi.org/10.1007/s12210-018-0756-x

WHAT IS X-RAY FLUORESCENCE?

Raw data

8 of 152

8

Portable XRF

Taccetti, F., Castelli, L., Czelusniak, C. et al. A multipurpose X-ray fluorescence scanner developed for in situ analysis. Rend. Fis. Acc. Lincei 30, 307–322 (2019). https://doi.org/10.1007/s12210-018-0756-x

the CHNet AIRES-CH project - Alessandro Bombini

Fe

Hg

Pb

Ca

Fe

WHAT IS X-RAY FLUORESCENCE?

9 of 152

9

  • It provides fast, sensitive, multi-elemental non-invasive, non-destructive analysis. It is perfect for Cultural Heritage applications;​
  • It can be performed with portable apparatus for in-situ analysis (e.g. in museums).​
  • Can produce macro-maps [~ O(meter)] (MA-XRF)​
  • It is able to detect signal coming from hidden pictorial layers, underneath the outermost one.​

WHY X-RAY FLUORESCENCE?

Visible Layer, Ragazzo Triste, XVIII sec. Circa, ​Unknown author​

XRF Sb image​, Ragazzo Triste, XVIII sec. Circa, ​Unknown author​

  • IT IS AVAILABLE TO US.

the CHNet AIRES-CH project - Alessandro Bombini

10 of 152

10

INTRINSIC LIMITS OF X-RAY FLUORESCENCE

the CHNet AIRES-CH project - Alessandro Bombini

  • Slow pace dataset creation (~O(10) XRF raw data / year)
  • Noisy data ( e.g. calibration ADC-to-Energy changes)
  • Data engineering needed (e.g. alignment/resizing of RGB pixels to XRF pixels)
  • Low detection limits of light elements
  • Blind to organic properties of pigments (they are always C-O-H compounds)

INTRINSIC LIMITS OF RGB

  • Non absolute color scale
  • Data engineering needed (e.g. alignment/resizing of RGB pixels to XRF pixels)
  • Color perception is subjective (it is a psychophysical phenomenon)
  • Pigments color is not absolute & changes in time (color degradation)

http://chemart.rice.edu/images/AnthraquinoneDyes.jpg

Are we doomed to fail?

(spoiler: no)

11 of 152

11

the CHNet AIRES-CH project - Alessandro Bombini

A NEW HOPE?

clustering MA-XRF raw data

12 of 152

12

the CHNet AIRES-CH project - Alessandro Bombini

A NEW HOPE?

clustering MA-XRF raw data

ENCODER

DECODER

CLUSTERING

ALGORITHM

Latent space

NOTE: NO DEEP LEARNING HERE

MA-XRF RAW DATA

MA-XRF clustered DATA

13 of 152

13

the CHNet AIRES-CH project - Alessandro Bombini

A NEW HOPE?

clustering MA-XRF raw data

PCA

PCA-1

CLUSTERING

ALGORITHM

Latent space

NOTE: NO DEEP LEARNING HERE

MA-XRF RAW DATA

MA-XRF clustered DATA

14 of 152

14

the CHNet AIRES-CH project - Alessandro Bombini

A NEW HOPE?

clustering MA-XRF raw data

15 of 152

15

the CHNet AIRES-CH project - Alessandro Bombini

A NEW HOPE?

clustering MA-XRF raw data

16 of 152

16

the CHNet AIRES-CH project - Alessandro Bombini

A NEW HOPE?

clustering MA-XRF raw data

17 of 152

17

the CHNet AIRES-CH project - Alessandro Bombini

A NEW HOPE?

clustering MA-XRF raw data

18 of 152

18

the CHNet AIRES-CH project - Alessandro Bombini

A NEW HOPE?

clustering MA-XRF raw data

19 of 152

19

the CHNet AIRES-CH project - Alessandro Bombini

A NEW HOPE?

​

  • MA-XRF raw data presents many hidden statistical relations we can exploit
  • We can either use peaks relations (i.e. 1D properties), or
  • We can use spatial segmentation (i.e. 2D properties)

A multidimensional neural network?

20 of 152

20

2D UNet-like

Refiner

Recolored Image

XRF

1D CNN

AIRES-CH

(Artificial Intelligence for digital REStoration of Cultural Heritage)

a.k.a. how to (try to) digitally restore damaged paintings using physics and AI

the CHNet AIRES-CH project - Alessandro Bombini

21 of 152

21

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

WHAT IS AIRES-CH?

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

XRF

2D UNet-like

1D CNN

Refiner

Recolored Image

22 of 152

22

XRF

2D UNet-like

1D CNN

Refiner

Recolored Image

  • Learns from millions of pixels’ histograms
  • Learns how to associate RGB to peaks distributions
  • Loses spatial correlations

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

WHAT IS AIRES-CH?

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

1D Branch

23 of 152

23

XRF

2D UNet-like

1D CNN

Refiner

Recolored Image

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

WHAT IS AIRES-CH?

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

  • Learns from few images
  • Learns how to associate RGB to regions AND peak distributions
  • Learns spatial correlations

2D Branch

24 of 152

24

XRF

2D UNet-like

1D CNN

Refiner

Recolored Image

  • Learns from few images but millions of pixels’ histograms
  • Learns how to associate RGB to regions AND peak distributions
  • Learns spatial correlations
  • The refiner networks learns how to properly merge the two

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

WHAT IS AIRES-CH?

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

Joint branches

25 of 152

25

  • Sometimes we face multi-layered pictorial artworks with hidden, older, no more visible layers, due to corrections, or pentimenti, or (more) modern restoration processes;
  • We may face damaged surfaces, especially of frescoes, where pigments are no more visible but still detectable using nuclear techniques.
  • A well-trained Deep Neural Network (DNN), capable of infer RGB from nuclear imaging raw data, may communicate something interesting about the pigment composition upon failing.
    1. Or, even, upon success.

WHY AIRES-CH?

the CHNet AIRES-CH project - Alessandro Bombini

It may be Lapis Lazuli Blue; probably it sees Ca in the Lazurite [(Na,Ca)8(AlSiO4)6(S,SO4,Cl)1–2] and K, sometimes substituting the Na. [https://www.mindat.org/min-2357.html]

Crocifissione di Viterbo, Unknown author, XVI sec. circa;

courtesy of Museum Polo Monumentale colle del Duomo, Viterbo & Archeoares srl.

26 of 152

26

the CHNet AIRES-CH project - Alessandro Bombini

Crocifissione di Viterbo, Unknown author, XVI sec. circa;

courtesy of Museum Polo Monumentale colle del Duomo, Viterbo & Archeoares srl.

It may be Lapis Lazuli Blue; probably it sees Ca in the Lazurite [(Na,Ca)8(AlSiO4)6(S,SO4,Cl)1–2] and K, sometimes substituting the Na. [https://www.mindat.org/min-2357.html]

WHY AIRES-CH?

it may be an overfitting feature. A good one, at least

27 of 152

27

AIRES-CH

Training Dataset

​

the CHNet AIRES-CH project - Alessandro Bombini

28 of 152

28

The whole dataset is composed by 62 XRF raw data coming from several XRF analysis on multiple paintings performed both in the LABEC facility in Florence, as well as in situ analysis (the data comes not only from published works, and include some private artworks).

For training the 1D models, only a 50% of the pixels where used (being randomly chosen), giving a training dataset of around 2,059,780 [histogram, RGB] pairs, divided into training, test, and validation set.

For training the 2D models, 45 XRF scans are used, reserving the remaining as 9 for test, and 8 for validation.

The raw data are obtained by three different devices, all developed, built and assembled by CHNet.

The raw data comes from different artwork typologies: multi-layered paintings, drawings without preparatory layers, and illuminated manuscripts, all over different periods and epochs (from middle ages to contemporary art).

It is worth noticing that artworks from different epochs might have been realised using different pigments. Visually similar colour can therefore be associated to completely different XRF spectra depending on the painting’s epoch.

TRAINING DATASET

the CHNet AIRES-CH project - Alessandro Bombini

Bombini, A., Anderlini, L., dell’Agnello, L., Giacomini, F., Ruberto, C., Taccetti, F.: The AIRES-CH project: Artificial Intelligence for digital REStoration of Cultural Heritages using physical imaging and multidimensional adversarial neural networks, Accepted for publication on the ICIAP2021 conference proceedings, Springer Lecture Notes in Computer Science vol. 13231

A huge thanks to the LABEC researchers for sharing their raw data to us:

​

Dr. Chiara Ruberto

Dr. Lisa Castelli

Dr. Anna Mazzinghi

29 of 152

29

AIRES-CH, first attempt

Neural Network Architectures

1D Branch

the CHNet AIRES-CH project - Alessandro Bombini

Alessandro Bombini, Lucio Anderlini, Luca dell’Agnello, Francesco Giaocmini, Chiara Ruberto, and Francesco Taccetti. 2022. The AIRES-CH Project: Artificial Intelligence for Digital REStoration of Cultural Heritages Using Nuclear Imaging and Multidimensional Neural Networks. In Image Analysis and Processing – ICIAP 2022: 21st International Conference, Lecce, Italy, May 23–27, 2022, Proceedings, Part I. Springer-Verlag, Berlin, Heidelberg, 685–700. https://doi.org/10.1007/978-3-031-06427-2_57

30 of 152

30

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

31 of 152

31

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

Perceptron

32 of 152

32

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

Multi-layer Perceptron (a.k.a. first Deep Neural Network)

Perceptron

33 of 152

33

the CHNet AIRES-CH project - Alessandro Bombini

TRAINING THE NET

Gradient Descent

34 of 152

34

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

Convolutional Neural Networks (CNN)

35 of 152

35

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

ResNet50 (2015)

Residual connections address Vanishing Gradient problem

36 of 152

36

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

Parallel, different-scale learning

Inception (2014)

Parallel, different-scale learning

37 of 152

37

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet
  • All of them were (moderately) capable of inferring the RGB from the XRF histogram.
    • Three of them were slightly more performing

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

38 of 152

38

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

v7_DilatedWaveNet

Input

Causal

Cinv1D

MaxPool1D

Dilated

Conv1D

BatchNorm

Multiply

Conv1D

Add

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Add

GlobalAveragePooling1D

  • Input Layer
  • CausalConv (kernel_size = 3)
  • MaxPool1D (2x2)
  • WaveNet Block
  • Conv1D (3)
  • DilatedConv1D (3, dil_rate = )
  • BatchNorm
  • Multiply

1 809 183 parameters

  • Add
  • GlobalAveragePooling1D
  • Sigmoid Activation
  • Tanh Activation
  • Dense
  • Dropout
  • Output

Oord, Aaron van den, et al. "Wavenet: A generative model for raw audio." arXiv preprint arXiv:1609.03499 (2016). https://doi.org/10.48550/arXiv.1609.03499

the CHNet AIRES-CH project - Alessandro Bombini

39 of 152

39

the CHNet AIRES-CH project - Alessandro Bombini

40 of 152

40

1D: v7_DilatedWaveNet

  • Input Layer
  • CausalConv (kernel_size = 3)
  • MaxPool1D (2x2)
  • WaveNet Block
  • Conv1D (3)
  • DilatedConv1D (3, dil_rate = )
  • BatchNorm
  • Multiply
  • Add
  • GlobalAveragePooling1D
  • Sigmoid Activation
  • Tanh Activation
  • Dense
  • Dropout
  • Output

Input

Causal

Cinv1D

MaxPool1D

Dilated

Conv1D

BatchNorm

Multiply

Conv1D

Add

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Add

GlobalAveragePooling1D

1 809 183 parameters

the CHNet AIRES-CH project - Alessandro Bombini

41 of 152

41

1D BRANCH MODELS

  • Input Layer (full hist)
  • MaxPool1D (2)
  • Fractal Block
  • Conv1D (kernel_size = 3)
  • BatchNorm
  • Dense
  • Dropout
  • Output
  • Internal block
  • Flatten

v6_FractalNet

2 117 195 parameters

Input

Input

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Flatten

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

InternalBlock

Conv1D

BatchNorm

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

Larsson, G., Maire, M., & Shakhnarovich, G. (2016). Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648. https://doi.org/10.48550/arXiv.1605.07648

the CHNet AIRES-CH project - Alessandro Bombini

42 of 152

42

the CHNet AIRES-CH project - Alessandro Bombini

43 of 152

43

1D: v6_FractalNet

  • Input Layer (full hist)
  • MaxPool1D (2)
  • Fractal Block
  • Conv1D (kernel_size = 3)
  • BatchNorm
  • Dense
  • Dropout
  • Output
  • Internal block
  • Flatten

2 117 195 parameters

Input

Input

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Flatten

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

InternalBlock

Conv1D

BatchNorm

the CHNet AIRES-CH project - Alessandro Bombini

44 of 152

44

1D BRANCH MODELS

why WaveNet & FractalNet?

Zabihi, M., Rad, A. B., Kiranyaz, S., Särkkä, S., & Gabbouj, M. (2019). 1d convolutional neural network models for sleep arousal detection. arXiv preprint arXiv:1903.01552.

https://doi.org/10.48550/arXiv.1903.01552

Overview of different 1D DNN for a completely different task

    • ResNet
    • FractalNet
    • WaveNet

the CHNet AIRES-CH project - Alessandro Bombini

45 of 152

45

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

1D: v5_CustomMultInputs

  • Input Layer (full hist)
  • Input Layer with normalization (bands)
  • MaxPool1D (2)
  • Generate 2-Grams
  • Conv1D (kernel_size = 3)
  • Concatenate
  • BatchNorm
  • Dense
  • Dropout
  • Output
  • ResBlock (reduce=True)
  • ResBlock (reduce=False)

.

.

.

.

.

.

Input

Conv1D

Conv1D

Input

Conv1D

Conv1D

Input

Conv1D

Conv1D

Generate 2-Grams

Conv1D

Conv1D

Conv1D

Conv1D

Conv1D

Conv1D

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

Concatenate

.

.

.

.

.

.

Input

MaxPool1D

ResBlock

ResBlock

ResBlock

Dense

Dropout

Output

Input

Conv1D

BatchNorm

Conv1D

Conv1D

MaxPool1D

BatchNorm

Conv1D

Conv1D

MaxPool1D

BatchNorm

Dense

Dropout

Dense

Dropout

Conv1D

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

2 482 731 parameters

Tin (Sn) Kα

Potassium (K) Kα

Manganese (Mn) Kα

Titanium (Ti) Kα

the CHNet AIRES-CH project - Alessandro Bombini

46 of 152

46

1D: v5_CustomMultInputs

  • Input Layer (full hist)
  • Input Layer with normalization (bands)
  • MaxPool1D (2)
  • Generate 2-Grams
  • Conv1D (kernel_size = 3)
  • Concatenate
  • BatchNorm
  • Dense
  • Dropout
  • Output
  • ResBlock (reduce=True)
  • ResBlock (reduce=False)

.

.

.

.

.

.

Input

Conv1D

Conv1D

Input

Conv1D

Conv1D

Input

Conv1D

Conv1D

Generate 2-Grams

Conv1D

Conv1D

Conv1D

Conv1D

Conv1D

Conv1D

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

Concatenate

.

.

.

.

.

.

Input

MaxPool1D

ResBlock

ResBlock

ResBlock

Dense

Dropout

Output

Input

Conv1D

BatchNorm

Conv1D

Conv1D

MaxPool1D

BatchNorm

Conv1D

Conv1D

MaxPool1D

BatchNorm

Dense

Dropout

Dense

Dropout

Conv1D

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

2 482 731 parameters

the CHNet AIRES-CH project - Alessandro Bombini

.

.

.

.

.

.

Relevant

subdominant

X lines

Sn (Kα)

K (Kα)

Mn (Kα)

Ti (Kα)

47 of 152

47

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

  • All of them were (moderately) capable of inferring the RGB from the XRF histogram.
    • Three of them were slightly more performing
  • We checked the performances using:
    • Structural Similarity Index Measure (SSIM)
    • Multi-Scale SSIM (MS-SSIM)
    • Peak Signal-to-Noise Ratio (PSNR)

the CHNet AIRES-CH project - Alessandro Bombini

48 of 152

48

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

  • All of them were (moderately) capable of inferring the RGB from the XRF histogram.
    • Three of them were slightly more performing
  • We checked the performances using:
    • Structural Similarity Index Measure (SSIM)
    • Multi-Scale SSIM (MS-SSIM)
    • Peak Signal-to-Noise Ratio (PSNR)

​

Binary Cross-Entropy

Mean Squared Error

v5_CustomMultInputs

0.636

0.0138

v6_FractalNet

0.633

0.0141

v7_WaveNet

0.629

0.0146

​

SSIM

MS-SSIM

PSNR

v5_CustomMultInputs

0.388

0.680

20.104

v6_FractalNet

0.372

0.677

20.076

v7_WaveNet

0.356

0.673

19.980

the CHNet AIRES-CH project - Alessandro Bombini

49 of 152

49

​

SSIM

MS-SSIM

PSNR

v5_CustomMultInputs

0.388

0.680

20.104

v6_FractalNet

0.372

0.677

20.076

v7_WaveNet

0.356

0.673

19.980

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

50 of 152

50

​

SSIM

MS-SSIM

PSNR

v5_CustomMultInputs

0.388

0.680

20.104

v6_FractalNet

0.372

0.677

20.076

v7_WaveNet

0.356

0.673

19.980

dynamic range

Structural Similarity (SSIM):

Zhou Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," in IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, April 2004, doi: 10.1109/TIP.2003.819861.

more on Visual Scores

51 of 152

51

Structural Similarity (SSIM):

​

SSIM

MS-SSIM

PSNR

v5_CustomMultInputs

0.388

0.680

20.104

v6_FractalNet

0.372

0.677

20.076

v7_WaveNet

0.356

0.673

19.980

Luminance

Contrast

Structure

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

52 of 152

52

​

SSIM

MS-SSIM

PSNR

v5_CustomMultInputs

0.388

0.680

20.104

v6_FractalNet

0.372

0.677

20.076

v7_WaveNet

0.356

0.673

19.980

Multiscale Structural Similarity (MS-SSIM):

more on Visual Scores

Z. Wang, E. P. Simoncelli and A. C. Bovik, "Multiscale structural similarity for image quality assessment," The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, 2003, pp. 1398-1402 Vol.2, doi: 10.1109/ACSSC.2003.1292216.

the CHNet AIRES-CH project - Alessandro Bombini

53 of 152

53

​

SSIM

MS-SSIM

PSNR

v5_CustomMultInputs

0.388

0.680

20.104

v6_FractalNet

0.372

0.677

20.076

v7_WaveNet

0.356

0.673

19.980

Peak Signal-to-Noise Ratio (PSNR):

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

54 of 152

54

​

SSIM

MS-SSIM

PSNR

v5_CustomMultInputs

0.388

0.680

20.104

v6_FractalNet

0.372

0.677

20.076

v7_WaveNet

0.356

0.673

19.980

Peak Signal-to-Noise Ratio (PSNR):

True Image

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

55 of 152

55

​

SSIM

MS-SSIM

PSNR

v5_CustomMultInputs

0.388

0.680

20.104

v6_FractalNet

0.372

0.677

20.076

v7_WaveNet

0.356

0.673

19.980

Peak Signal-to-Noise Ratio (PSNR):

Signal

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

56 of 152

56

​

SSIM

MS-SSIM

PSNR

v5_CustomMultInputs

0.388

0.680

20.104

v6_FractalNet

0.372

0.677

20.076

v7_WaveNet

0.356

0.673

19.980

Peak Signal-to-Noise Ratio (PSNR):

Noise

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

57 of 152

57

more on Visual Scores

Damon M. Chandler, "Seven Challenges in Image Quality Assessment: Past, Present, and Future Research", International Scholarly Research Notices, vol. 2013, Article ID 905685, 53 pages, 2013. https://doi.org/10.1155/2013/905685

the CHNet AIRES-CH project - Alessandro Bombini

58 of 152

58

  • We have developed and trained few DNN models:
    • Dense
    • CNN
    • ResNet-like
    • Inception-like
    • Custom Model
    • FractalNet
    • (Dilated)WaveNet

1D BRANCH MODELS

  • All of them were (moderately) capable of inferring the RGB from the XRF histogram.
    • Three of them were slightly more performing
  • We checked the performances using:
    • Structural Similarity Index Measure (SSIM)
    • Multi-Scale SSIM (MS-SSIM)
    • Peak Signal-to-Noise Ratio (PSNR)

​

Binary Cross-Entropy

Mean Squared Error

v5_CustomMultInputs

0.636

0.0138

v6_FractalNet

0.633

0.0141

v7_WaveNet

0.629

0.0146

​

SSIM

MS-SSIM

PSNR

v5_CustomMultInputs

0.388

0.680

20.104

v6_FractalNet

0.372

0.677

20.076

v7_WaveNet

0.356

0.673

19.980

the CHNet AIRES-CH project - Alessandro Bombini

59 of 152

59

EXAMPLE ON VALIDATION RAW DATA - multilayered painting

GOOD

the CHNet AIRES-CH project - Alessandro Bombini

60 of 152

60

EXAMPLE ON VALIDATION RAW DATA - multilayered painting

GOOD

the CHNet AIRES-CH project - Alessandro Bombini

61 of 152

61

EXAMPLE ON VALIDATION RAW DATA - illuminated manuscripts

NOT GOOD

the CHNet AIRES-CH project - Alessandro Bombini

62 of 152

62

EXAMPLE ON VALIDATION RAW DATA - illuminated manuscripts

NOT GOOD

​

  • Pentagram drawn in the back page;

XRF sees behind outermost pictorial layer

​

  • too many layers are difficult to recolor

the CHNet AIRES-CH project - Alessandro Bombini

63 of 152

63

AIRES-CH NN Architectures

Hyperparameter optimisation

1D Branch

Bombini, A., Anderlini, L., dell’Agnello, L., Giacomini, F., Ruberto, C., Taccetti, F.: Hyperparameter optimisation of Artificial Intelligence for digital REStoration of Cultural Heritages (AIRES-CH) models, In: Computational Science and Its Applications – ICCSA 2022 Workshops. ICCSA 2022. Lecture Notes in Computer Science, vol 13377. Springer, Cham. https://doi.org/10.1007/978-3-031-10536-4_7

the CHNet AIRES-CH project - Alessandro Bombini

64 of 152

64

Starting from the same dataset employed in the first steps, we decided to employ Optuna to fine-tune the DNN hyperparameters.

In particular, we exploited the Optuna ability of explore categorical parameter space to build different genres of layers.

The optimisation procedure was performed over 1000 trials; each training was performed over a 150-epochs, but with the caveat of invoking a callback for imposing an early stopping condition by monitoring the validation loss, to avoid overfitting.

On top of that, Optuna imposes its pruning condition to abort unpromising trials.

​

​

​

​

Note on loss choice: we use the binary cross-entropy as loss, because is the same loss employed before.

HYPERPARAMETER OPTIMISATION w/ OPTUNA

the CHNet AIRES-CH project - Alessandro Bombini

For a nice introduction on hyperparameter optimisation with Optuna, see Dr. Matteo Barbetti’s talk @ 3° ML_INFN Hackhaton

​

https://agenda.infn.it/event/32568/contributions/180769/attachments/99647/138397/bayes_hp_opt-3rd_mlinfn_hackathon.pdf

65 of 152

65

Starting from the same dataset employed in the first steps, we decided to employ Optuna to fine-tune the DNN hyperparameters.

In particular, we exploited the Optuna ability of explore categorical parameter space to build different genres of layers.

The optimisation procedure was performed over 1000 trials; each training was performed over a 150-epochs, but with the caveat of invoking a callback for imposing an early stopping condition by monitoring the validation loss, to avoid overfitting.

On top of that, Optuna imposes its pruning condition to abort unpromising trials.

​

​

​

​

Note on loss choice: we use the binary cross-entropy as loss, because is the same loss employed before.

HYPERPARAMETER OPTIMISATION w/ OPTUNA

the CHNet AIRES-CH project - Alessandro Bombini

For a nice introduction on hyperparameter optimisation with Optuna, see Dr. Matteo Barbetti’s talk @ 3° ML_INFN Hackhaton

​

https://agenda.infn.it/event/32568/contributions/180769/attachments/99647/138397/bayes_hp_opt-3rd_mlinfn_hackathon.pdf

66 of 152

66

For the 1D branch, we explore a variety of models’ composition by join subsequent, different parts, each of which picked by Optuna:

​

​

​

HYPERPARAMETER OPTIMISATION w/ OPTUNA

Conv

Conv

. . .

. . .

Dense

Dropout

Flatten

Dense

Dropout

. . .

Output

Input

Conv Part

(Conv1D)

Middle Part

(Inception, DilRes)

Dense Part

(Dense, Dropout)

the CHNet AIRES-CH project - Alessandro Bombini

67 of 152

67

For the 1D branch, we explore a variety of models’ composition by join subsequent, different parts, each of which picked by Optuna:

​

​

​

​

  1. the first part may comprise various Conv1D networks, with various kernel sizes and number of filters, and may comprise a Batch Normalisation layer;
  2. The second part is formed by a series of either Inception block or Dilated Residual block, their kernel size, and their number of filters. For the dilated residual blocks, their dilation rate is picked by Optuna;
  3. the third part is a series of couples Dense+Dropout; optuna picks the number of dense nodes, the dropout percentage and the activation function, chosen from the list [’relu’, ’sigmoid’, ’tanh’, ’softsign’, ’selu’].

HYPERPARAMETER OPTIMISATION w/ OPTUNA

Conv

Conv

. . .

. . .

Dense

Dropout

Flatten

Dense

Dropout

. . .

Output

Input

Conv Part

(Conv1D)

Middle Part

(Inception, DilRes)

Dense Part

(Dense, Dropout)

the CHNet AIRES-CH project - Alessandro Bombini

68 of 152

68

The Optuna results were:

​

​

​

​

HYPERPARAMETER OPTIMISATION w/ OPTUNA:

RESULTS

Conv

Conv

. . .

. . .

Dense

Dropout

Flatten

Dense

Dropout

. . .

Output

Input

Conv Part

(Conv1D)

Middle Part

(Inception, DilRes)

Dense Part

(Dense, Dropout)

  1. Input layer;
  2. Conv1D, K = 6 + ReLU activation;
  3. Conv1D, K = 6 + BatchNorm + ReLU activation;
  4. Conv1D, K = 6 + BatchNorm + ReLU activation;
  5. Conv1D, K = 5 + ReLU activation;

Conv Part

the CHNet AIRES-CH project - Alessandro Bombini

69 of 152

69

The Optuna results were:

​

​

​

​

HYPERPARAMETER OPTIMISATION w/ OPTUNA:

RESULTS

Conv

Conv

. . .

. . .

Dense

Dropout

Flatten

Dense

Dropout

. . .

Output

Input

Conv Part

(Conv1D)

Middle Part

(Inception, DilRes)

Dense Part

(Dense, Dropout)

  • Input layer;
  • Conv1D, K = 6 + ReLU activation;
  • Conv1D, K = 6 + BatchNorm + ReLU activation;
  • Conv1D, K = 6 + BatchNorm + ReLU activation;
  • Conv1D, K = 5 + ReLU activation;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;

Conv Part

Middle Part

the CHNet AIRES-CH project - Alessandro Bombini

70 of 152

70

The Optuna results were:

​

​

​

​

HYPERPARAMETER OPTIMISATION w/ OPTUNA:

RESULTS

Conv

Conv

. . .

. . .

Dense

Dropout

Flatten

Dense

Dropout

. . .

Output

Input

Conv Part

(Conv1D)

Middle Part

(Inception, DilRes)

Dense Part

(Dense, Dropout)

  • Input layer;
  • Conv1D, K = 6 + ReLU activation;
  • Conv1D, K = 6 + BatchNorm + ReLU activation;
  • Conv1D, K = 6 + BatchNorm + ReLU activation;
  • Conv1D, K = 5 + ReLU activation;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Average Pooling 1D, pool size = 6;
  • Dropout layer, α = 0.43;
  • Flatten layers;
  • Dense + selu activation;
  • Dropout layer, α = 0.39;
  • Dense + sigmoid activation;
  • Dropout layer, α = 0.23;
  • Dense + sigmoid activation;
  • Output Dense layer;

Conv Part

Middle Part

Dense Part

the CHNet AIRES-CH project - Alessandro Bombini

71 of 152

71

The Optuna results were:

​

​

​

​

HYPERPARAMETER OPTIMISATION w/ OPTUNA:

RESULTS

Conv

Conv

. . .

. . .

Dense

Dropout

Flatten

Dense

Dropout

. . .

Output

Input

Conv Part

(Conv1D)

Middle Part

(Inception, DilRes)

Dense Part

(Dense, Dropout)

  • Input layer;
  • Conv1D, K = 6 + ReLU activation;
  • Conv1D, K = 6 + BatchNorm + ReLU activation;
  • Conv1D, K = 6 + BatchNorm + ReLU activation;
  • Conv1D, K = 5 + ReLU activation;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Average Pooling 1D, pool size = 6;
  • Dropout layer, α = 0.43;
  • Flatten layers;
  • Dense + selu activation;
  • Dropout layer, α = 0.39;
  • Dense + sigmoid activation;
  • Dropout layer, α = 0.23;
  • Dense + sigmoid activation;
  • Output Dense layer;

Conv Part

Middle Part

Dense Part

the CHNet AIRES-CH project - Alessandro Bombini

72 of 152

72

The Optuna results were:

​

​

​

​

HYPERPARAMETER OPTIMISATION w/ OPTUNA:

RESULTS

Conv

Conv

. . .

. . .

Dense

Dropout

Flatten

Dense

Dropout

. . .

Output

Input

Conv Part

(Conv1D)

Middle Part

(Inception, DilRes)

Dense Part

(Dense, Dropout)

  • Input layer;
  • Conv1D, K = 6 + ReLU activation;
  • Conv1D, K = 6 + BatchNorm + ReLU activation;
  • Conv1D, K = 6 + BatchNorm + ReLU activation;
  • Conv1D, K = 5 + ReLU activation;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Inception block;
  • Average Pooling 1D, pool size = 6;
  • Dropout layer, α = 0.43;
  • Flatten layers;
  • Dense + selu activation;
  • Dropout layer, α = 0.39;
  • Dense + sigmoid activation;
  • Dropout layer, α = 0.23;
  • Dense + sigmoid activation;
  • Output Dense layer;

Conv Part

Middle Part

Dense Part

the CHNet AIRES-CH project - Alessandro Bombini

73 of 152

73

AIRES-CH, first attempt

Neural Network Architectures

2D Branch

Alessandro Bombini, Lucio Anderlini, Luca dell’Agnello, Francesco Giaocmini, Chiara Ruberto, and Francesco Taccetti. 2022. The AIRES-CH Project: Artificial Intelligence for Digital REStoration of Cultural Heritages Using Nuclear Imaging and Multidimensional Neural Networks. In Image Analysis and Processing – ICIAP 2022: 21st International Conference, Lecce, Italy, May 23–27, 2022, Proceedings, Part I. Springer-Verlag, Berlin, Heidelberg, 685–700. https://doi.org/10.1007/978-3-031-06427-2_57

the CHNet AIRES-CH project - Alessandro Bombini

74 of 152

74

  • Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:
    • VGG-like
    • DilResNet-like

2D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

75 of 152

75

  • Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:
    • VGG-like
    • DilResNet-like

2D BRANCH MODELS

Encoder

VGG-19

Decoder

VGG-19T

  • Input Layer
  • Conv2D (3x3)
  • MaxPool2D (2x2)
  • Dilated Residual Block
  • Conv2DT (2x2)
  • Concatenate+Conv2D (3x3)
  • Output Layer

2D: VGG UNet

15 246 659 parameters

the CHNet AIRES-CH project - Alessandro Bombini

76 of 152

76

  • Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:
    • VGG-like
    • DilResNet-like

2D BRANCH MODELS

Encoder

Decoder

DilResBlocks

  • Input Layer
  • Conv2D (3x3)
  • MaxPool2D (2x2)
  • Dilated Residual Block
  • Conv2DT (2x2)
  • Concatenate+Conv2D (3x3)
  • Output Layer

2D: Dilated Residual UNet

2 179 779 parameters

the CHNet AIRES-CH project - Alessandro Bombini

77 of 152

77

  • Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:
    • VGG-like
    • DilResNet-like

2D BRANCH MODELS

  • Both were, again, (moderately) capable of inferring the RGB from the XRF histogram.

the CHNet AIRES-CH project - Alessandro Bombini

78 of 152

78

  • Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:
    • VGG-like
    • DilResNet-like

2D BRANCH MODELS

  • Both were, again, (moderately) capable of inferring the RGB from the XRF histogram.

​

Binary Cross-Entropy

Mean Squared Error

VGG

0.480

0.0159

DilResNet

0.460

0.0121

​

SSIM

MS-SSIM

PSNR

VGG

0.733

0.626

19.691

DilResNet

0.745

0.669

21.097

the CHNet AIRES-CH project - Alessandro Bombini

79 of 152

79

  • Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:
    • VGG-like
    • DilResNet-like

2D BRANCH MODELS

  • Both were, again, (moderately) capable of inferring the RGB from the XRF histogram.

​

Binary Cross-Entropy

Mean Squared Error

VGG

0.480

0.0159

DilResNet

0.460

0.0121

​

SSIM

MS-SSIM

PSNR

VGG

0.733

0.626

19.691

DilResNet

0.745

0.669

21.097

the CHNet AIRES-CH project - Alessandro Bombini

80 of 152

80

  • Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:
    • VGG-like
    • DilResNet-like

2D BRANCH MODELS

  • Both were, again, (moderately) capable of inferring the RGB from the XRF histogram.

​

Binary Cross-Entropy

Mean Squared Error

VGG

0.480

0.0159

DilResNet

0.460

0.0121

​

SSIM

MS-SSIM

PSNR

VGG

0.733

0.626

19.691

DilResNet

0.745

0.669

21.097

    • The best performing model out of the two is the DilResNet model.
    • They seems to outperform 1D models on each score/metric BUT MS-SSIM

the CHNet AIRES-CH project - Alessandro Bombini

81 of 152

81

EXAMPLE ON VALIDATION RAW DATA - multilayered painting

GOOD

2D

1D

(best model)

the CHNet AIRES-CH project - Alessandro Bombini

82 of 152

82

EXAMPLE ON VALIDATION RAW DATA - multilayered painting

GOOD

2D

1D

(best model)

the CHNet AIRES-CH project - Alessandro Bombini

83 of 152

83

EXAMPLE ON VALIDATION RAW DATA - multilayered painting

GOOD

MS-SSIM best score

2D

1D

(best model)

the CHNet AIRES-CH project - Alessandro Bombini

84 of 152

84

EXAMPLE ON VALIDATION RAW DATA - illuminated manuscripts

NOT GOOD

Here 2D outperforms 1D

2D

1D

(best model)

the CHNet AIRES-CH project - Alessandro Bombini

85 of 152

85

AIRES-CH NN Architectures

Hyperparameter optimisation

2D Branch

Bombini, A., Anderlini, L., dell’Agnello, L., Giacomini, F., Ruberto, C., Taccetti, F.: Hyperparameter optimisation of Artificial Intelligence for digital REStoration of Cultural Heritages (AIRES-CH) models, In: Computational Science and Its Applications – ICCSA 2022 Workshops. ICCSA 2022. Lecture Notes in Computer Science, vol 13377. Springer, Cham. https://doi.org/10.1007/978-3-031-10536-4_7

the CHNet AIRES-CH project - Alessandro Bombini

86 of 152

86

HYPERPARAMETER OPTIMISATION w/ OPTUNA

Conv2D (3x3)

. . .

. . .

. . .

RGB

Decoder

(Conv2D, Conv2DTranspose)

Middle

(Inception, DilRes)

Encoder

(Conv2D, MaxPool)

For the 2D branch, we explore a variety of UNet-like models’ composition by join subsequent, different parts, each of which picked by Optuna:

​

​

​

the CHNet AIRES-CH project - Alessandro Bombini

87 of 152

87

Conv2D (3x3)

. . .

. . .

. . .

RGB

Decoder

(Conv2D, Conv2DTranspose)

Middle

(Inception, DilRes)

Encoder

(Conv2D, MaxPool)

For the 2D branch, we explore a variety of UNet-like models’ composition by join subsequent, different parts, each of which picked by Optuna:

​

​

​

HYPERPARAMETER OPTIMISATION w/ OPTUNA

1. The number of Convolutional block layers in the Encoder/Decoder, and each of their kernel sizes and the number of filters;

2. The number of internal blocks; for each internal block, its type (either Inception block or Dilated Residual block), its kernel size, and its number of filters. For the dilated residual block(s), its dilation rate.

the CHNet AIRES-CH project - Alessandro Bombini

88 of 152

88

Conv2D (3x3)

. . .

. . .

. . .

RGB

Decoder

(Conv2D, Conv2DTranspose)

Middle

(Inception, DilRes)

Encoder

(Conv2D, MaxPool)

For the 2D branch, we explore a variety of UNet-like models’ composition by join subsequent, different parts, each of which picked by Optuna:

​

​

​

HYPERPARAMETER OPTIMISATION w/ OPTUNA

1. The number of Convolutional block layers in the Encoder/Decoder, and each of their kernel sizes and the number of filters;

2. The number of internal blocks; for each internal block, its type (either Inception block or Dilated Residual block), its kernel size, and its number of filters. For the dilated residual block(s), its dilation rate.

​

Unfortunately, due to hardware limitations, we were not able to perform the training using the whole 500-channels energy depth (as in the first attempts);

​

To reduce the in-memory size of the XRF tensors, we extracted the (supposedly) 26-most relevant elemental maps [see the paper for more details].

​

Nevertheless, we did not expect this to come without issues: sub-dominant peaks, as well as sub-dominant elements, may be as much important as those elements, and the coarse-graining of the input data may vastly impact the scores of the model.

Technical Issue faced:

the CHNet AIRES-CH project - Alessandro Bombini

89 of 152

89

1. Conv Block, K = (1, 1) + Conv2DT (2, 2);

2. Conv Block, K = (2, 2) + Conv2DT (2, 2);

3. Conv Block, K = (4, 4) + Conv2DT (2, 2);

4. Conv Block, K = (3, 3) + Conv2DT (2, 2);

5. Conv Block, K = (5, 5) + Conv2DT (2, 2);

6. Output Conv2D layer, K = (3, 3)

Decoder

1. Input layer;

2. Conv Block, K = (5, 5) + MaxPool (2, 2);

3. Conv Block, K = (3, 3) + MaxPool (2, 2);

4. Conv Block, K = (4, 4) + MaxPool (2, 2);

5. Conv Block, K = (2, 2) + MaxPool (2, 2);

6. Conv Block, K = (1, 1) + MaxPool (2, 2);

Encoder

HYPERPARAMETER OPTIMISATION w/ OPTUNA

RESULTS

Conv2D (3x3)

. . .

. . .

. . .

RGB

Decoder

(Conv2D, Conv2DTranspose)

Middle

(Inception, DilRes)

Encoder

(Conv2D, MaxPool)

For the 2D branch, the results are:

the CHNet AIRES-CH project - Alessandro Bombini

90 of 152

90

1. Input layer;

2. Conv Block, K = (5, 5) + MaxPool (2, 2);

3. Conv Block, K = (3, 3) + MaxPool (2, 2);

4. Conv Block, K = (4, 4) + MaxPool (2, 2);

5. Conv Block, K = (2, 2) + MaxPool (2, 2);

6. Conv Block, K = (1, 1) + MaxPool (2, 2);

Encoder

1. Conv Block, K = (1, 1) + Conv2DT (2, 2);

2. Conv Block, K = (2, 2) + Conv2DT (2, 2);

3. Conv Block, K = (4, 4) + Conv2DT (2, 2);

4. Conv Block, K = (3, 3) + Conv2DT (2, 2);

5. Conv Block, K = (5, 5) + Conv2DT (2, 2);

6. Output Conv2D layer, K = (3, 3)

Decoder

1. Inception block;

2. Inception block;

3. DilResNet block, K = (4, 4), d = 4;

4. DilResNet block, K = (5, 5), d = 2;

5. DilResNet block, K = (5, 5), d = 4;

6. DilResNet block, K = (5, 5), d = 4;

Middle

HYPERPARAMETER OPTIMISATION w/ OPTUNA

RESULTS

For the 2D branch, the results are:

Conv2D (3x3)

. . .

. . .

. . .

RGB

Decoder

(Conv2D, Conv2DTranspose)

Middle

(Inception, DilRes)

Encoder

(Conv2D, MaxPool)

the CHNet AIRES-CH project - Alessandro Bombini

91 of 152

91

1. Input layer;

2. Conv Block, K = (5, 5) + MaxPool (2, 2);

3. Conv Block, K = (3, 3) + MaxPool (2, 2);

4. Conv Block, K = (4, 4) + MaxPool (2, 2);

5. Conv Block, K = (2, 2) + MaxPool (2, 2);

6. Conv Block, K = (1, 1) + MaxPool (2, 2);

Encoder

1. Conv Block, K = (1, 1) + Conv2DT (2, 2);

2. Conv Block, K = (2, 2) + Conv2DT (2, 2);

3. Conv Block, K = (4, 4) + Conv2DT (2, 2);

4. Conv Block, K = (3, 3) + Conv2DT (2, 2);

5. Conv Block, K = (5, 5) + Conv2DT (2, 2);

6. Output Conv2D layer, K = (3, 3)

Decoder

1. Inception block;

2. Inception block;

3. DilResNet block, K = (4, 4), d = 4;

4. DilResNet block, K = (5, 5), d = 2;

5. DilResNet block, K = (5, 5), d = 4;

6. DilResNet block, K = (5, 5), d = 4;

Middle

HYPERPARAMETER OPTIMISATION w/ OPTUNA

RESULTS

For the 2D branch, the results are:

Conv2D (3x3)

. . .

. . .

. . .

RGB

Decoder

(Conv2D, Conv2DTranspose)

Middle

(Inception, DilRes)

Encoder

(Conv2D, MaxPool)

the CHNet AIRES-CH project - Alessandro Bombini

92 of 152

92

1. Input layer;

2. Conv Block, K = (5, 5) + MaxPool (2, 2);

3. Conv Block, K = (3, 3) + MaxPool (2, 2);

4. Conv Block, K = (4, 4) + MaxPool (2, 2);

5. Conv Block, K = (2, 2) + MaxPool (2, 2);

6. Conv Block, K = (1, 1) + MaxPool (2, 2);

Encoder

1. Conv Block, K = (1, 1) + Conv2DT (2, 2);

2. Conv Block, K = (2, 2) + Conv2DT (2, 2);

3. Conv Block, K = (4, 4) + Conv2DT (2, 2);

4. Conv Block, K = (3, 3) + Conv2DT (2, 2);

5. Conv Block, K = (5, 5) + Conv2DT (2, 2);

6. Output Conv2D layer, K = (3, 3)

Decoder

1. Inception block;

2. Inception block;

3. DilResNet block, K = (4, 4), d = 4;

4. DilResNet block, K = (5, 5), d = 2;

5. DilResNet block, K = (5, 5), d = 4;

6. DilResNet block, K = (5, 5), d = 4;

Middle

HYPERPARAMETER OPTIMISATION w/ OPTUNA

RESULTS

For the 2D branch, the results are:

Conv2D (3x3)

. . .

. . .

. . .

RGB

Decoder

(Conv2D, Conv2DTranspose)

Middle

(Inception, DilRes)

Encoder

(Conv2D, MaxPool)

the CHNet AIRES-CH project - Alessandro Bombini

93 of 152

93

AIRES-CH

Results

to opt or not to opt?

the CHNet AIRES-CH project - Alessandro Bombini

94 of 152

94

RESULTS:

  1. For the 1D Branch:

​

Binary Cross-Entropy

Mean Squared Error

1D Opt

0.659

0.0119

1D non-Opt

0.623

0.0119

​

SSIM

MS-SSIM

PSNR

1D Opt

0.459

0.739

19.524

1D non-Opt

0.388

0.680

20.104

Performances of the 1D Opt model vs best non-Opt model.

The best scores are written in red

the CHNet AIRES-CH project - Alessandro Bombini

95 of 152

95

RESULTS:

  • For the 1D Branch:

​

Binary Cross-Entropy

Mean Squared Error

1D Opt

0.659

0.0119

1D non-Opt

0.623

0.0119

​

SSIM

MS-SSIM

PSNR

1D Opt

0.459

0.739

19.524

1D non-Opt

0.388

0.680

20.104

Performances of the 1D Opt model vs best non-Opt model.

The best scores are written in red

Visual comparison on the results of the 1D models;

the true RGB image is shown on the left, the prediction of old best model on the middle, and the prediction of the optimised model on the right.

the CHNet AIRES-CH project - Alessandro Bombini

96 of 152

96

RESULTS:

  • For the 1D Branch:
    • We see that it slightly outperforms the best model of the first attempt in most scores, especially in the one found in that work as the most relevant, the MS-SSIM;
    • From the visual result, we can see a small improvement in the average appearance, especially on the dark tones, as the SSIM and MS-SSIM scores tells us;
    • nevertheless, it seems that, even if a little bit noisy, the old model seems to be more capable of detecting rare pigment specimen, as the PSNR score seems to suggest.

​

Binary Cross-Entropy

Mean Squared Error

1D Opt

0.659

0.0119

1D non-Opt

0.623

0.0128

​

SSIM

MS-SSIM

PSNR

1D Opt

0.459

0.739

19.524

1D non-Opt

0.388

0.680

20.104

Performances of the 1D Opt model vs best non-Opt model.

The best scores are written in red

the CHNet AIRES-CH project - Alessandro Bombini

97 of 152

97

RESULTS:

​

Binary Cross-Entropy

Mean Squared Error

2D non-Opt

0.4604

0.0121

2D Opt

0.4950

0.0230

​

SSIM

MS-SSIM

PSNR

2D non-Opt

0.745

0.669

21.097

2D Opt

0.659

0.648

17.420

  • For the 1D Branch:
    • We see that it slightly outperforms the best model of the first attempt in most scores, especially in the one found in that work as the most relevant, the MS-SSIM;
    • From the visual result, we can see a small improvement in the average appearance, especially on the dark tones, as the SSIM and MS-SSIM scores tells us;
    • nevertheless, it seems that, even if a little bit noisy, the old model seems to be more capable of detecting rare pigment specimen, as the PSNR score seems to suggest.
  • For the 2D Branch:
    • Unfortunately, due to the issues described before, the performances of the model, were slightly lower w.r.t. the ones of the old best 2D model;

Performances of the 2D Opt model vs best non-Opt model.

The best scores are written in red

the CHNet AIRES-CH project - Alessandro Bombini

98 of 152

98

RESULTS:

​

Binary Cross-Entropy

Mean Squared Error

2D non-Opt

0.4604

0.0121

2D Opt

0.4950

0.0230

​

SSIM

MS-SSIM

PSNR

2D non-Opt

0.745

0.669

21.097

2D Opt

0.659

0.648

17.420

  • For the 1D Branch:
    • We see that it slightly outperforms the best model of the first attempt in most scores, especially in the one found in that work as the most relevant, the MS-SSIM;
    • From the visual result, we can see a small improvement in the average appearance, especially on the dark tones, as the SSIM and MS-SSIM scores tells us;
    • nevertheless, it seems that, even if a little bit noisy, the old model seems to be more capable of detecting rare pigment specimen, as the PSNR score seems to suggest.
  • For the 2D Branch:
    • Unfortunately, due to the issues described before, the performances of the model, were slightly lower w.r.t. the ones of the old best 2D model;

Performances of the 2D Opt model vs best non-Opt model.

The best scores are written in red

​

This seems to mean that subdominant peaks + background contribution is crucial to correctly interpret the XRF raw data.

​

AS EXPECTED [from XRF theory]!

​

But this also implies that we can dramatically improve the hyperparameter optimisation of the 2D models, simply by improving the hardware set up!

Results of the technical Issue faced:

the CHNet AIRES-CH project - Alessandro Bombini

99 of 152

99

RESULTS RECAP:

​

Binary Cross-Entropy

Mean Squared Error

1D Opt

0.659

0.0119

2D Opt

0.4950

0.0230

​

SSIM

MS-SSIM

PSNR

1D Opt

0.459

0.739

19.524

2D Opt

0.659

0.648

17.420

  • For the 1D Branch:
    • We see that it slightly outperforms the best model of the first attempt in most scores, especially in the one found in that work as the most relevant, the MS-SSIM;
    • From the visual result, we can see a small improvement in the average appearance, especially on the dark tones, as the SSIM and MS-SSIM scores tells us;
    • nevertheless, it seems that, even if a little bit noisy, the old model seems to be more capable of detecting rare pigment specimen, as the PSNR score seems to suggest.
  • For the 2D Branch:
    • Unfortunately, due to the issues described before, the performances of the model, were slightly lower w.r.t. the ones of the old best 2D model;

Performances of the Optimised models.

The scores where the optimised

model over-performs the respective old best-models are written in red

the CHNet AIRES-CH project - Alessandro Bombini

100 of 152

100

deploying AIRES-CH models

A cloud-native application for digital restoration of Cultural Heritage using nuclear imaging

THESPIAN-XRF

the CHNet AIRES-CH project - Alessandro Bombini

101 of 152

101

Web softwares for data analysis: XRF Analyser

the CHNet AIRES-CH project - Alessandro Bombini

Dash is a Open Source Python library for creating reactive, Web-based applications. it’s entirely open-source and MIT licensed. Dash app code is declarative and reactive, which makes it easy to build complex apps that contain many interactive elements.

​

https://github.com/plotly/dash

Flask is a Python web framework built with a small core and easy-to-extend philosophy.

​

It is easy-to-use, and is also easy to get started with as a beginner because there is little boilerplate code for getting a simple app up and running.

https://github.com/pallets/flask

M. Grinberg, The Flask mega tutorial, https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world

the stack

102 of 152

102

Web softwares for data analysis: XRF Analyser

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

​

User may easily visualize the XRF image with few clicks.

the CHNet AIRES-CH project - Alessandro Bombini

103 of 152

103

Web softwares for data analysis: XRF Analyser

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

​

User may easily visualize the XRF image with few clicks.

the CHNet AIRES-CH project - Alessandro Bombini

104 of 152

104

Web softwares for data analysis: XRF Analyser

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

​

User may easily visualize the XRF image with few clicks.

the CHNet AIRES-CH project - Alessandro Bombini

105 of 152

105

Web softwares for data analysis: XRF Analyser

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

​

User may easily visualize the XRF image with few clicks.

the CHNet AIRES-CH project - Alessandro Bombini

106 of 152

106

Web softwares for data analysis: XRF Analyser

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

​

User may easily visualize the XRF image with few clicks.

the CHNet AIRES-CH project - Alessandro Bombini

107 of 152

107

Web softwares for data analysis: XRF Analyser

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

​

User may easily visualize the XRF image with few clicks.

the CHNet AIRES-CH project - Alessandro Bombini

108 of 152

108

Web softwares for data analysis: XRF Analyser

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

​

User may easily visualize the XRF image with few clicks.

the CHNet AIRES-CH project - Alessandro Bombini

109 of 152

109

Web softwares for data analysis: XRF Analyser

the CHNet AIRES-CH project - Alessandro Bombini

110 of 152

110

Web softwares for data analysis: XRF Analyser

Advanced visualisation tools - 3D Plot

the CHNet AIRES-CH project - Alessandro Bombini

111 of 152

111

Web softwares for data analysis: XRF Analyser

Advanced visualisation tools - ROI rgb image

R: 1° Element

G: 2° Element

B: 3° Element

the CHNet AIRES-CH project - Alessandro Bombini

112 of 152

112

Embedding AIRES-CH in the cloud

Web softwares for data analysis: XRF Analyser

the CHNet AIRES-CH project - Alessandro Bombini

113 of 152

113

the CHNet AIRES-CH project - Alessandro Bombini

Embedding AIRES-CH in the cloud

Web softwares for data analysis: XRF Analyser

  • Accurate spatial segmentation

​

  • Few correct colour association

114 of 152

114

the CHNet AIRES-CH project - Alessandro Bombini

Embedding AIRES-CH in the cloud

Web softwares for data analysis: XRF Analyser

  • Accurate spatial segmentation

​

  • Few correct colour association

115 of 152

115

the CHNet AIRES-CH project - Alessandro Bombini

Embedding AIRES-CH in the cloud

Web softwares for data analysis: XRF Analyser

  • Seems an unsatisfactory result…
  • But:
    • It is a raw data of a new analysis (i.e. different historical period w.r.t. training dataset)
    • More importantly, it is a different artwork typology!

And it still works! (even if poorly)

116 of 152

116

Embedding AIRES-CH in the cloud

Web softwares for data analysis: XRF Analyser

the CHNet AIRES-CH project - Alessandro Bombini

THESPIAN

XRF

XRF

​

read-only

  1. Gunicorn is used for deploying the App as Web Gateway Server Interface (WSGI)
  • The application is containerised using Docker

Docker can package an application and its dependencies in a virtual container that can run on any Os. This enables the application to run in a variety of locations, such as on-premises, in public or private cloud.

When running on Linux, Docker uses the resource isolation features of the Linux kernel and a union-capable file system to allow containers to run within a single Linux instance, avoiding the overhead

of starting and maintaining

virtual machines.

easy deployment!

117 of 152

117

Embedding AIRES-CH in the cloud

Web softwares for data analysis: XRF Analyser

the CHNet AIRES-CH project - Alessandro Bombini

1

6

Reverse proxy

Application

Indigo-IAM

OPA

2

3

3.2a

3.1a

4

5

3.1b

3.2b

THESPIAN

XRF

XRF

​

read-only

Internal

Application

Federated Application

THESPIAN

Mask

Front End

​

Back End

​

DB

​

118 of 152

118

Embedding AIRES-CH in the cloud

Web softwares for data analysis: XRF Analyser

the CHNet AIRES-CH project - Alessandro Bombini

Reverse

Proxy

Front End

Back End

DataBase

​

​

THESPIAN

Mask

Front End

​

Back End

​

DB

​

THESPIAN-XRF

XRF

​

- Dash

(almost)

micro

service

architecture

119 of 152

119

AIRES-CH

Conclusions

what’s next?

the CHNet AIRES-CH project - Alessandro Bombini

120 of 152

120

the CHNet AIRES-CH project - Alessandro Bombini

  • We have shown that the goal of inferring an RGB image from an XRF raw data is feasible.
    • We have developed both 1D & 2D branches
  • We have planned a new measurement campaign, jointly with Biblioteca Marucelliana, Firenze, on their drawings, to enlarge and standardise the training dataset.
  • We have started using Optuna to perform hyperparameter optimisation of the models.
    • It [slightly] improved the 1D branch;
  • We have embedded an alpha version on the network inside our XRF web tool

CONCLUSIONS & OUTLOOK

121 of 152

121

  • We have shown that the goal of inferring an RGB image from an XRF raw data is feasible.
    • We have developed both 1D & 2D branches
  • We have planned a new measurement campaign, jointly with Biblioteca Marucelliana, Firenze, on their drawings, to enlarge and standardise the training dataset.
  • We have started using Optuna to perform hyperparameter optimisation of the models.
    • It [slightly] improved the 1D branch;

the CHNet AIRES-CH project - Alessandro Bombini

CONCLUSIONS & OUTLOOK

MACHINA

XRF

World Wide Web

raw data

Cloud

  • We have embedded an alpha version on the network inside our XRF web tool
  • We are going to develop the technique to take into account the presence of hidden pictorial layers and to recolour them, somehow factoring out the contribution from the outermost layer ⇒ ganX project
  • This is not the only application: we can use Internet Of Things (IoT) and link to the web the imaging instruments (especially the portable ones for in situ analysis, like XRF and MACHINA), so that we can send to server the raw data to have real-time recoloring.

122 of 152

122

the CHNet AIRES-CH project - Alessandro Bombini

A special thanks to…

the AIRES squad

Dr. Chiara Ruberto,

responsible for XRF Analysis of the AIRES-CH project.

Conceptualisation, Data collection, Data Analysis, use and upgrade of the XRF apparatus;

Dr. Fernando García-Avello Bofías,

assistant researcher of the AIRES-CH project.

Data collection, Investigation, Methodology, Software development, Neural Network Design and training;

Dr. Francesco Taccetti,

PI of the AIRES-CH project.

Conceptualisation, funding acquisition, project administration,

XRF development

Collaborators

  • Dr. Lucio Anderlini (ML_INFN)

​

​

  • Dr. Luca dell’Agnello (CNAF)
  • Dr. Francesco Giacomini (CNAF)

123 of 152

Alessandro Bombini

email: bombini@fi.infn.it

Thank you for your attention!

AIRES-CH paper 1

AIRES-CH paper 2

124 of 152

Alessandro Bombini

email: bombini@fi.infn.it

Backup slides

AIRES-CH paper 1

AIRES-CH paper 2

125 of 152

125

125

Before AIRES-CH

Cultural Heritage Network

how it is organised

the CHNet AIRES-CH project - Alessandro Bombini

126 of 152

126

1st level nodes:�Laboratories in INFN facilities

2nd level nodes:�Universities, Restoration Centres, Associations, with complementary competencies

3rd level nodes:�Foreign research centres/Universities outside Europe.

- Each 3rd level node is bound to a 1st level node

- Each 3rd level node is encouraged to create a �local network with different competencies in its own country

the CHNet AIRES-CH project - Alessandro Bombini

Research�(50%)

External Projects

INFN-CHNet

INFN

Education�(20%)

Third Mission�(30%)

Access to �Laboratories and� Instrumentation�(30%)

Public� Engagement�(40%)

Technology�Transfer�(30%)

The INFN

Cultural Heritage Network

Giuntini, L.; Castelli, L.; Massi, M.; Fedi, M.; Czelusniak, C.; Gelli, N.; Liccioli, L.; Giambi, F.; Ruberto, C.; Mazzinghi, A.; Barone, S.; Marchegiani, F.; Nisi, S.; Lubritto, C.; Altieri, S.; Tortora, L.; Branchini, P.; Fabbri, A.; Graziani, V.; Lins, S.B.; Guidorzi, L.; Lo Giudice, A.; Re, A.; Sottili, L.; Balerna, A.; Cestelli Guidi, M.; Pronti, L.; Romani, M.; Albertin, F.; Bettuzzi, M.; Brancaccio, R.; Morigi, M.P.; Alloni, D.; Salvini, A.; Smilgys, B.; Prata, M.; Altieri, S.; Bonesini, M.; Di Martino, D.; Clemenza, M.; Carpinelli, M.; Oliva, P.; Sipala, V.; Gueli, A.M.; Pasquale, S.; Stella, G.; Pepponi, G.; Grazzi, F.; Taccetti, F. Detectors and Cultural Heritage: The INFN-CHNet Experience. Appl. Sci. 2021, 11, 3462. https://doi.org/10.3390/app11083462

127 of 152

127

127

the CHNet AIRES-CH project - Alessandro Bombini

The INFN-CHNET

Digital Heritage Laboratory

INFN-CHNet is organised in three research domain:

  1. FixLab: the ensemble of immovable apparatus for in-lab analysis
  2. MoLab: the ensemble of movable apparatus for in-situ analysis
  3. DHLab: the ensemble of digital services for research.

The DHLab (Digital Heritage Laboratory) is in charge of developing software service for the INFN-CHNet researchers, (mostly of which) to be offered on the CHNet cloud,

mainly in the EU Projects Ariadne+, EOSC-Pillar and 4CH.

​

It currently offers cloud services developed within 2 european projects, EOSC-Pillar and Ariadne+.

In the near future, they will be integrated into the 4CH competence center on Cultural Heritage,

where INFN is task leader of the Tasks 3.3 (Cultural Heritage Cloud) and 3.5 (Big Data Services).

​

​

​

It is also involved in digital technology research.

CHNet cloud

Bombini, A., Castelli, L., dell’Agnello, L., Felicetti, A., Giacomini, F., Niccolucci, F., Taccetti, F.: CHNet cloud: an EOSC-based cloud for physical technologies applied

to cultural heritages. In: GARR (ed.) Conferenza GARR 2021 - Sostenibile/Digitale.

Dati e tecnologie per il futuro.

(2021). https://doi.org/10.26314/GARR-Conf21-proceedings-09,

128 of 152

128

the CHNet AIRES-CH project - Alessandro Bombini

MACHINA

(Movable Accelerator for Cultural Heritage In-situ Non-destructive Analysis)

S. Mathot, G. Anelli, S. Atieh, A. Bilton, B. Bulat, Th. Callamand, S. Calvo, G. Favre, J.-M. Geisser, A. Gerardin, A. Grudiev, A. Lombardi, E. Montesinos, F. Motschmann, H. Pommerenke, P. Richerot, K. Scibor, M. Timmins, M. Vretenar, F. Taccetti, F. Benetti, L. Castelli, M. Chiari, C. Czelusniak, S. Falciano, M. Fedi, P.A. Mandò, M. Manetti, C. Matacotta, E. Previtali, C. Ruberto, V. Virgili, L. Giuntini, The CERN PIXE-RFQ, a transportable proton accelerator for the machina project, Nuclear Instruments and Methods in Physics Research Section B, https://doi.org/10.1016/j.nimb.2019.08.025

Portable XRF

Taccetti, F., Castelli, L., Czelusniak, C. et al. A multipurpose X-ray fluorescence scanner developed for in situ analysis. Rend. Fis. Acc. Lincei 30, 307–322 (2019). https://doi.org/10.1007/s12210-018-0756-x

3MV Tandetron

14C AMS line

PIXE-PIGE line

Chiari, M., et al. LABEC, the INFN ion beam laboratory of nuclear techniques for environment and cultural heritage. Eur. Phys. J. Plus 136, 472 (2021). https://doi.org/10.1140/epjp/s13360-021-01411-1

The INFN-CHNet Florence node

LABEC

129 of 152

129

Monolith vs Microservices vs MicroFrontEnds

A small note on Application Architecture

THESPIAN-XRF

the CHNet AIRES-CH project - Alessandro Bombini

130 of 152

130

the CHNet AIRES-CH project - Alessandro Bombini

SPA

SPA

Monolith

Back End

Monolith

Front End

Back End

DataBase

Micro

services

131 of 152

131

the CHNet AIRES-CH project - Alessandro Bombini

SPA

SPA

Monolith

Back End

Monolith

Front End

Back End

DataBase

Micro

services

https://semaphoreci.com/blog/microfrontends

Micro

Front

Ends

With a microfrontend, no single team owns the UI in its entirety. Instead, every team owns a piece of the screen, page, or content. For example, one team might be responsible for the search box, while another might code suggestions based on users’ tastes. Additional teams might code the music player, manage playlists, or render the billing page. We add complexity but teams get increased autonomy in return.

132 of 152

132

literature review

Deep Learning on Physical Imaging data

Hyperspectral analysis

the CHNet AIRES-CH project - Alessandro Bombini

133 of 152

133

(re)using data: AI, ML & DNN Applications

C. Cucci, J. K. Delaney, and M. Picollo, Reflectance Hyperspectral Imaging for Investigation of Works of Art: Old Master Paintings and Illuminated Manuscripts, Accounts of Chemical Research 2016 49 (10), 2070-2079 https://doi.org/10.1021/acs.accounts.6b00048

134 of 152

134

(re)using data: AI, ML & DNN Applications

C. Cucci, J. K. Delaney, and M. Picollo, Reflectance Hyperspectral Imaging for Investigation of Works of Art: Old Master Paintings and Illuminated Manuscripts, Accounts of Chemical Research 2016 49 (10), 2070-2079 https://doi.org/10.1021/acs.accounts.6b00048

135 of 152

135

(re)using data: AI, ML & DNN Applications

C. Cucci, J. K. Delaney, and M. Picollo, Reflectance Hyperspectral Imaging for Investigation of Works of Art: Old Master Paintings and Illuminated Manuscripts, Accounts of Chemical Research 2016 49 (10), 2070-2079 https://doi.org/10.1021/acs.accounts.6b00048

136 of 152

136

(re)using data: AI, ML & DNN Applications

Spectral Angular Mapper (SAM)

Hyperspectral

Kleynhans, T., Schmidt Patterson, C.M., Dooley, K.A. et al. An alternative approach to mapping pigments in paintings with hyperspectral reflectance image cubes using artificial intelligence. Herit Sci 8, 84 (2020). https://doi.org/10.1186/s40494-020-00427-7

137 of 152

137

(re)using data: AI, ML & DNN Applications

Spectral Angular Mapper (SAM)

Mapping

Hyperspectral

False colors

Kleynhans, T., Schmidt Patterson, C.M., Dooley, K.A. et al. An alternative approach to mapping pigments in paintings with hyperspectral reflectance image cubes using artificial intelligence. Herit Sci 8, 84 (2020). https://doi.org/10.1186/s40494-020-00427-7

138 of 152

138

(re)using data: AI, ML & DNN Applications

Spectral Angular Mapper (SAM)

Mapping

Hyperspectral

False colors

Kleynhans, T., Schmidt Patterson, C.M., Dooley, K.A. et al. An alternative approach to mapping pigments in paintings with hyperspectral reflectance image cubes using artificial intelligence. Herit Sci 8, 84 (2020). https://doi.org/10.1186/s40494-020-00427-7

139 of 152

139

(re)using data: AI, ML & DNN Applications

Convolutional Neural Networks (CNN)

Kleynhans, T., Schmidt Patterson, C.M., Dooley, K.A. et al. An alternative approach to mapping pigments in paintings with hyperspectral reflectance image cubes using artificial intelligence. Herit Sci 8, 84 (2020). https://doi.org/10.1186/s40494-020-00427-7

140 of 152

140

(re)using data: AI, ML & DNN Applications

Convolutional Neural Networks (CNN)

Kleynhans, T., Schmidt Patterson, C.M., Dooley, K.A. et al. An alternative approach to mapping pigments in paintings with hyperspectral reflectance image cubes using artificial intelligence. Herit Sci 8, 84 (2020). https://doi.org/10.1186/s40494-020-00427-7

141 of 152

141

(re)using data: AI, ML & DNN Applications

Convolutional Neural Networks (CNN)

Kleynhans, T., Schmidt Patterson, C.M., Dooley, K.A. et al. An alternative approach to mapping pigments in paintings with hyperspectral reflectance image cubes using artificial intelligence. Herit Sci 8, 84 (2020). https://doi.org/10.1186/s40494-020-00427-7

142 of 152

142

(re)using data: AI, ML & DNN Applications

Convolutional Neural Networks (CNN)

Kleynhans, T., Schmidt Patterson, C.M., Dooley, K.A. et al. An alternative approach to mapping pigments in paintings with hyperspectral reflectance image cubes using artificial intelligence. Herit Sci 8, 84 (2020). https://doi.org/10.1186/s40494-020-00427-7

143 of 152

143

(re)using data: AI, ML & DNN Applications

Convolutional Neural Networks (CNN)

Kleynhans, T., Schmidt Patterson, C.M., Dooley, K.A. et al. An alternative approach to mapping pigments in paintings with hyperspectral reflectance image cubes using artificial intelligence. Herit Sci 8, 84 (2020). https://doi.org/10.1186/s40494-020-00427-7

The spectra of the brown ochre class collected from all four paintings showing the spectral variability. The black dotted line is the average spectrum for brown ochre

INPUTS

CLASSES

144 of 152

144

pre AIRES-CH

Clustering MA-XRF raw data

using PCA for high dimensionality reduction & clustering

the CHNet AIRES-CH project - Alessandro Bombini

145 of 152

145

the CHNet AIRES-CH project - Alessandro Bombini

MORE ON:

clustering MA-XRF raw data

ENCODER

DECODER

CLUSTERING

ALGORITHM

Latent space

NOTE: NO DEEP LEARNING HERE

MA-XRF RAW DATA

MA-XRF clustered DATA

146 of 152

146

the CHNet AIRES-CH project - Alessandro Bombini

MORE ON:

clustering MA-XRF raw data

MA-XRF RAW DATA

ENCODER

V. Satopaa, J. Albrecht, D. Irwin and B. Raghavan, "Finding a "Kneedle" in a Haystack: Detecting Knee Points in System Behavior," 2011 31st International Conference on Distributed Computing Systems Workshops, 2011, pp. 166-171, doi: 10.1109/ICDCSW.2011.20.

147 of 152

147

the CHNet AIRES-CH project - Alessandro Bombini

MORE ON:

clustering MA-XRF raw data

DECODER

CLUSTERING

ALGORITHM

Latent space

ENCODER

148 of 152

Alessandro Bombini

email: bombini@fi.infn.it

Extra Backup

AIRES-CH paper 1

AIRES-CH paper 2

149 of 152

149

Web softwares for data analysis: XRF Analyser

the CHNet AIRES-CH project - Alessandro Bombini

150 of 152

150

Web softwares for data analysis: XRF Analyser

Advanced visualisation tools

the CHNet AIRES-CH project - Alessandro Bombini

151 of 152

151

Web softwares for data analysis: XRF Analyser

Advanced visualisation tools

the CHNet AIRES-CH project - Alessandro Bombini

152 of 152

152

152

the CHNet AIRES-CH project - Alessandro Bombini

STRATIGRAPHY OF PICTORIAL ARTWORKS

support

glue (+ chalk) ground

white priming

preparatory drawing

wood, canvas, etc.

organic glues, etc.

Pb white, Ca white, etc.

graphite

painted layer 1

multi-elemental pigments

painted layer n

multi-elemental pigments

varnish

organic