1 of 96

Using X-ray technologies and deep neural networks for digital restoration of paintings: the CHNet AIRES-CH project

Alessandro Bombini, INFN Firenze

AI@INFN, Bologna, May 3rd, 2022

2 of 96

2

02-03 May 2022 AI@INFN

2D UNet-like

Refiner

Recolored Image

XRF

1D CNN

AIRES-CH

(Artificial Intelligence for digital REStoration of Cultural Heritage)

a.k.a. how to (try to) digitally restore damaged paintings using physics and AI

the CHNet AIRES-CH project - Alessandro Bombini

Bombini, A., Anderlini, L., dell’Agnello, L., Giacomini, F., Ruberto, C., Taccetti, F.: The AIRES-CH project: Artificial Intelligence for digital REStoration of Cultural Heritages using physical imaging and multidimensional adversarial neural networks, Accepted for publication on the ICIAP2021 conference proceedings, Springer Lecture Notes in Computer Science vol. 13231

…STARTING SOON…

3 of 96

3

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

WHAT IS AIRES-CH?

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

4 of 96

4

02-03 May 2022 AI@INFN

XRF

2D UNet-like

1D CNN

Refiner

Recolored Image

Learns from millions of pixels’ histograms
Learns how to associate RGB to peaks distributions
Loses spatial correlations

the CHNet AIRES-CH project - Alessandro Bombini

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

WHAT IS AIRES-CH?

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

5 of 96

5

02-03 May 2022 AI@INFN

XRF

2D UNet-like

1D CNN

Refiner

Recolored Image

the CHNet AIRES-CH project - Alessandro Bombini

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

WHAT IS AIRES-CH?

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

Learns from few images
Learns how to associate RGB to regions AND peak distributions
Learns spatial correlations

6 of 96

6

02-03 May 2022 AI@INFN

XRF

2D UNet-like

1D CNN

Refiner

Recolored Image

Learns from few images but millions of pixels’ histograms
Learns how to associate RGB to regions AND peak distributions
Learns spatial correlations
The refiner networks learns how to properly merge the two

the CHNet AIRES-CH project - Alessandro Bombini

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

WHAT IS AIRES-CH?

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

7 of 96

7

02-03 May 2022 AI@INFN

Sometimes we face multi-layered pictorial artworks with hidden, older, no more visible layers, due to corrections, or pentimenti, or (more) modern restoration processes;
We may face damaged surfaces, especially of frescoes, where pigments are no more visible but still detectable using nuclear techniques.
A well-trained Deep Neural Network (DNN), capable of infer RGB from nuclear imaging raw data, may communicate something interesting about the pigment composition upon failing.

Or, even, upon success.

WHY AIRES-CH?

the CHNet AIRES-CH project - Alessandro Bombini

It may be Lapis Lazuli Blue; probably it sees Ca in the Lazurite [(Na,Ca)₈(AlSiO₄)₆(S,SO₄,Cl)_1–2] and K, sometimes substituting the Na. [https://www.mindat.org/min-2357.html]

Crocifissione di Viterbo, Unknown author, XVI sec. circa;

courtesy of Museum Polo Monumentale colle del Duomo, Viterbo & Archeoares srl.

8 of 96

8

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

Portable XRF

Taccetti, F., Castelli, L., Czelusniak, C. et al. A multipurpose X-ray fluorescence scanner developed for in situ analysis. Rend. Fis. Acc. Lincei 30, 307–322 (2019). https://doi.org/10.1007/s12210-018-0756-x

WHAT IS X-RAY FLUORESCENCE?

9 of 96

9

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

Portable XRF

Taccetti, F., Castelli, L., Czelusniak, C. et al. A multipurpose X-ray fluorescence scanner developed for in situ analysis. Rend. Fis. Acc. Lincei 30, 307–322 (2019). https://doi.org/10.1007/s12210-018-0756-x

Fe

Hg

PB

Ca

Fe

XRF visualiser

WHAT IS X-RAY FLUORESCENCE?

10 of 96

10

02-03 May 2022 AI@INFN

It provides fast, sensitive, multi-elemental non-invasive, non-destructive analysis. It is perfect for Cultural Heritage applications;
It can be performed with portable apparatus for in-situ analysis (e.g. in museums).
Can produce macro-maps [~ O(meter)]
It is able to detect signal coming from hidden pictorial layers, underneath the outermost one.

WHY X-RAY FLUORESCENCE?

Visible Layer, Ragazzo Triste, XVIII sec. Circa, Unknown author

XRF Sb image, Ragazzo Triste, XVIII sec. Circa, Unknown author

IT IS AVAILABLE TO US.

the CHNet AIRES-CH project - Alessandro Bombini

11 of 96

11

02-03 May 2022 AI@INFN

AIRES-CH

Training Dataset

the CHNet AIRES-CH project - Alessandro Bombini

12 of 96

12

02-03 May 2022 AI@INFN

The whole dataset is composed by 62 XRF raw data coming from several XRF analysis on multiple paintings performed both in the LABEC facility in Florence, as well as in situ analysis (the data comes not only from published works, and include some private artworks).

For training the 1D models, only a 50% of the pixels where used (being randomly chosen), giving a training dataset of around 2,059,780 [histogram, RGB] pairs, divided into training, test, and validation set.

For training the 2D models, 45 XRF scans are used, reserving the remaining as 9 for test, and 8 for validation.

The raw data are obtained by three different devices, all developed, built and assembled by CHNet.

The raw data comes from different artwork typologies: multi-layered paintings, drawings without preparatory layers, and illuminated manuscripts, all over different periods and epochs (from middle ages to contemporary art).

It is worth noticing that artworks from different epochs might have been realised using different pigments. Visually similar colour can therefore be associated to completely different XRF spectra depending on the painting’s epoch.

TRAINING DATASET

the CHNet AIRES-CH project - Alessandro Bombini

Bombini, A., Anderlini, L., dell’Agnello, L., Giacomini, F., Ruberto, C., Taccetti, F.: The AIRES-CH project: Artificial Intelligence for digital REStoration of Cultural Heritages using physical imaging and multidimensional adversarial neural networks, Accepted for publication on the ICIAP2021 conference proceedings, Springer Lecture Notes in Computer Science vol. 13231

13 of 96

13

02-03 May 2022 AI@INFN

AIRES-CH

Neural Network Architectures

1D Branch

the CHNet AIRES-CH project - Alessandro Bombini

14 of 96

14

02-03 May 2022 AI@INFN

We have developed and trained few DNN models:

Dense
CNN
ResNet-like
Inception-like
Custom Model
FractalNet
(Dilated)WaveNet

All of them were (moderately) capable of inferring the RGB from the XRF histogram.

Three of them were slightly more performing

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

15 of 96

15

02-03 May 2022 AI@INFN

We have developed and trained few DNN models:

Dense
CNN
ResNet-like
Inception-like
Custom Model
FractalNet
(Dilated)WaveNet

All of them were (moderately) capable of inferring the RGB from the XRF histogram.

Three of them were slightly more performing

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

16 of 96

16

02-03 May 2022 AI@INFN

We have developed and trained few DNN models:

Dense
CNN
ResNet-like
Inception-like
Custom Model
FractalNet
(Dilated)WaveNet

1D BRANCH MODELS

v7_DilatedWaveNet

Input

Causal

Cinv1D

MaxPool1D

Dilated

Conv1D

BatchNorm

Multiply

Conv1D

Add

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Add

GlobalAveragePooling1D

Input Layer
CausalConv (kernel_size = 3)
MaxPool1D (2x2)
WaveNet Block
Conv1D (3)
DilatedConv1D (3, dil_rate = )
BatchNorm
Multiply

1 809 183 parameters

Add
GlobalAveragePooling1D
Sigmoid Activation
Tanh Activation
Dense
Dropout
Output

17 of 96

17

02-03 May 2022 AI@INFN

1D BRANCH MODELS

Input Layer (full hist)
MaxPool1D (2)
Fractal Block
Conv1D (kernel_size = 3)
BatchNorm

Dense
Dropout
Output
Internal block
Flatten

v6_FractalNet

2 117 195 parameters

Input

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Flatten

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

Conv1D

BatchNorm

We have developed and trained few DNN models:

Dense
CNN
ResNet-like
Inception-like
Custom Model
FractalNet
(Dilated)WaveNet

18 of 96

18

02-03 May 2022 AI@INFN

We have developed and trained few DNN models:

Dense
CNN
ResNet-like
Inception-like
Custom Model
FractalNet
(Dilated)WaveNet

1D BRANCH MODELS

1D: v5_CustomMultInputs

Input Layer (full hist)
Input Layer with normalization (bands)
MaxPool1D (2)
Generate 2-Grams
Conv1D (kernel_size = 3)
Concatenate
BatchNorm

Dense
Dropout
Output
ResBlock (reduce=True)
ResBlock (reduce=False)

.

Input

Conv1D

Input

Conv1D

Input

Conv1D

Generate 2-Grams

Conv1D

ResBlock

Concatenate

.

Input

MaxPool1D

ResBlock

Dense

Dropout

Output

Input

Conv1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Dense

Dropout

Dense

Dropout

Conv1D

ResBlock

2 482 731 parameters

Tin (Sn) Kα

Potassium (K) Kα

Manganese (Mn) Kα

Titanium (Ti) Kα

19 of 96

19

02-03 May 2022 AI@INFN

We have developed and trained few DNN models:

Dense
CNN
ResNet-like
Inception-like
Custom Model
FractalNet
(Dilated)WaveNet

1D BRANCH MODELS

All of them were (moderately) capable of inferring the RGB from the XRF histogram.

Three of them were slightly more performing

20 of 96

20

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

EXAMPLE ON VALIDATION RAW DATA - multilayered painting

GOOD

21 of 96

21

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

EXAMPLE ON VALIDATION RAW DATA - multilayered painting

GOOD

22 of 96

22

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

EXAMPLE ON VALIDATION RAW DATA - illuminated manuscripts

NOT GOOD

Pentagram drawn in the back page;

XRF sees behind outermost pictorial layer

too many layers are difficult to recolor

23 of 96

23

02-03 May 2022 AI@INFN

AIRES-CH

Neural Network Architectures

2D Branch

the CHNet AIRES-CH project - Alessandro Bombini

24 of 96

24

02-03 May 2022 AI@INFN

Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:

VGG-like
DilResNet-like

2D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

25 of 96

25

02-03 May 2022 AI@INFN

Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:

VGG-like
DilResNet-like

2D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

Encoder

VGG-19

Decoder

VGG-19^T

Input Layer
Conv2D (3x3)
MaxPool2D (2x2)
Dilated Residual Block
Conv2DT (2x2)
Concatenate+Conv2D (3x3)
Output Layer

2D: VGG UNet

15 246 659 parameters

26 of 96

26

02-03 May 2022 AI@INFN

Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:

VGG-like
DilResNet-like

2D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

Encoder

Decoder

DilResBlocks

Input Layer
Conv2D (3x3)
MaxPool2D (2x2)
Dilated Residual Block
Conv2DT (2x2)
Concatenate+Conv2D (3x3)
Output Layer

2D: Dilated Residual UNet

2 179 779 parameters

27 of 96

27

02-03 May 2022 AI@INFN

Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:

VGG-like
DilResNet-like

2D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

Both were, again, (moderately) capable of inferring the RGB from the XRF histogram.

	Binary Cross-Entropy	Mean Squared Error
VGG	0.480	0.0159
DilResNet	0.460	0.0121

	SSIM	MS-SSIM	PSNR
VGG	0.733	0.626	19.691
DilResNet	0.745	0.669	21.097

The best performing model out of the two is the DilResNet model.

They seems to outperform 1D models on each score/metric BUT MS-SSIM

28 of 96

28

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

EXAMPLE ON VALIDATION RAW DATA - multilayered painting

GOOD

MS-SSIM best score

2D

1D

(best model)

29 of 96

29

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

EXAMPLE ON VALIDATION RAW DATA - illuminated manuscripts

NOT GOOD

Here 2D outperforms 1D

2D

1D

(best model)

30 of 96

30

02-03 May 2022 AI@INFN

AIRES-CH

Conclusions

Outlook

the CHNet AIRES-CH project - Alessandro Bombini

31 of 96

31

We have started using Optuna to perform hyperparameter optimisation of the models.

02-03 May 2022 AI@INFN

We have shown that the goal of inferring an RGB image from an XRF raw data is feasible.

We have developed both 1D & 2D branches

We have planned a new measurement campaign, jointly with Biblioteca Marucelliana, Firenze, on their drawings, to enlarge and standardise the training dataset.

CONCLUSIONS & OUTLOOK

the CHNet AIRES-CH project - Alessandro Bombini

MACHINA

XRF

World Wide Web

raw data

Cloud

Bombini, A., Anderlini, L., dell’Agnello, L., Giacomini, F., Ruberto, C., Taccetti, F.: Hyperparameter optimisation of Artificial Intelligence for digital REStoration of Cultural Heritages (AIRES-CH) models, Under review Workshop on Advancements in Applied Machine-learning and Data Analytics (AAMDA), Springer Lecture Notes in Computer Science

We have embedded an alpha version on the network inside our XRF web tool
We are going to develop the technique to take into account the presence of hidden pictorial layers and to recolour them, somehow factoring out the contribution from the outermost layer.
This is not the only application: we can use Internet Of Things (IoT) and link to the web the imaging instruments (especially the portable ones for in situ analysis, like XRF and MACHINA), so that we can send to server the raw data to have real-time recoloring.

32 of 96

Alessandro Bombini

email: bombini@fi.infn.it

Thank you for your attention!

33 of 96

Extra slides

Alessandro Bombini

email: bombini@fi.infn.it

34 of 96

34

02-03 May 2022 AI@INFN

INFN-CHNet

Cultural Heritage Network

how it is organised

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

35 of 96

35

1^st level nodes:�Laboratories in INFN facilities

2^nd level nodes:�Universities, Restoration Centres, Associations, with complementary competencies

3^rd level nodes:�Foreign research centres/Universities outside Europe.

- Each 3^rd level node is bound to a 1^st level node

- Each 3^rd level node is encouraged to create a �local network with different competencies in its own country

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

Research�(50%)

External Projects

INFN-CHNet

INFN

Education�(20%)

Third Mission�(30%)

Access to �Laboratories and� Instrumentation�(30%)

Public� Engagement�(40%)

Technology�Transfer�(30%)

The INFN

Cultural Heritage Network

Giuntini, L.; Castelli, L.; Massi, M.; Fedi, M.; Czelusniak, C.; Gelli, N.; Liccioli, L.; Giambi, F.; Ruberto, C.; Mazzinghi, A.; Barone, S.; Marchegiani, F.; Nisi, S.; Lubritto, C.; Altieri, S.; Tortora, L.; Branchini, P.; Fabbri, A.; Graziani, V.; Lins, S.B.; Guidorzi, L.; Lo Giudice, A.; Re, A.; Sottili, L.; Balerna, A.; Cestelli Guidi, M.; Pronti, L.; Romani, M.; Albertin, F.; Bettuzzi, M.; Brancaccio, R.; Morigi, M.P.; Alloni, D.; Salvini, A.; Smilgys, B.; Prata, M.; Altieri, S.; Bonesini, M.; Di Martino, D.; Clemenza, M.; Carpinelli, M.; Oliva, P.; Sipala, V.; Gueli, A.M.; Pasquale, S.; Stella, G.; Pepponi, G.; Grazzi, F.; Taccetti, F. Detectors and Cultural Heritage: The INFN-CHNet Experience. Appl. Sci. 2021, 11, 3462. https://doi.org/10.3390/app11083462

GO BACK

36 of 96

36

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

The INFN-CHNET

Digital Heritage Laboratory

INFN-CHNet is organised in three research domain:

FixLab: the ensemble of immovable apparatus for in-lab analysis
MoLab: the ensemble of movable apparatus for in-situ analysis
DHLab: the ensemble of digital services for research.

The DHLab (Digital Heritage Laboratory) is in charge of developing software service for the INFN-CHNet researchers, (mostly of which) to be offered on the CHNet cloud,

mainly in the EU Projects Ariadne+, EOSC-Pillar and 4CH.

It currently offers cloud services developed within 2 european projects, EOSC-Pillar and Ariadne+.

In the near future, they will be integrated into the 4CH competence center on Cultural Heritage,

where INFN is task leader of the Tasks 3.3 (Cultural Heritage Cloud) and 3.5 (Big Data Services).

It is also involved in digital technology research.

CHNet cloud

Bombini, A., Castelli, L., dell’Agnello, L., Felicetti, A., Giacomini, F., Niccolucci, F., Taccetti, F.: CHNet cloud: an EOSC-based cloud for physical technologies applied

to cultural heritages. In: GARR (ed.) Conferenza GARR 2021 - Sostenibile/Digitale.

Dati e tecnologie per il futuro.

(2021). https://doi.org/10.26314/GARR-Conf21-proceedings-09,

GO BACK

37 of 96

37

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

MACHINA

(Movable Accelerator for Cultural Heritage In-situ Non-destructive Analysis)

S. Mathot, G. Anelli, S. Atieh, A. Bilton, B. Bulat, Th. Callamand, S. Calvo, G. Favre, J.-M. Geisser, A. Gerardin, A. Grudiev, A. Lombardi, E. Montesinos, F. Motschmann, H. Pommerenke, P. Richerot, K. Scibor, M. Timmins, M. Vretenar, F. Taccetti, F. Benetti, L. Castelli, M. Chiari, C. Czelusniak, S. Falciano, M. Fedi, P.A. Mandò, M. Manetti, C. Matacotta, E. Previtali, C. Ruberto, V. Virgili, L. Giuntini, The CERN PIXE-RFQ, a transportable proton accelerator for the machina project, Nuclear Instruments and Methods in Physics Research Section B, https://doi.org/10.1016/j.nimb.2019.08.025

Portable XRF

Taccetti, F., Castelli, L., Czelusniak, C. et al. A multipurpose X-ray fluorescence scanner developed for in situ analysis. Rend. Fis. Acc. Lincei 30, 307–322 (2019). https://doi.org/10.1007/s12210-018-0756-x

3MV Tandetron

¹⁴C AMS line

PIXE-PIGE line

Chiari, M., et al. LABEC, the INFN ion beam laboratory of nuclear techniques for environment and cultural heritage. Eur. Phys. J. Plus 136, 472 (2021). https://doi.org/10.1140/epjp/s13360-021-01411-1

The INFN-CHNet Florence node

LABEC

GO BACK

38 of 96

38

02-03 May 2022 AI@INFN

DHLAB tools

XRF visualizer

how it works

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

39 of 96

39

The XRF (X-Ray Fluorescence) is a physical imaging technology, which allows researchers to explore the elemental composition of a study object;

Applied to pictoric image, such a painting or a fresco, or to a document, such miniated manuscripts, allows us to understand the elemental composition of pigments.

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

User may easily visualize the XRF image with few clicks.

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

40 of 96

40

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

User may easily visualize the XRF image with few clicks.

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

The XRF (X-Ray Fluorescence) is a physical imaging technology, which allows researchers to explore the elemental composition of a study object;

Applied to pictoric image, such a painting or a fresco, or to a document, such miniated manuscripts, allows us to understand the elemental composition of pigments.

GO BACK

41 of 96

41

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

The XRF (X-Ray Fluorescence) is a physical imaging technology, which allows researchers to explore the elemental composition of a study object;

Applied to pictoric image, such a painting or a fresco, or to a document, such miniated manuscripts, allows us to understand the elemental composition of pigments.

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

User may easily visualize the XRF image with few clicks.

GO BACK

42 of 96

42

Kα Fe

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

The XRF (X-Ray Fluorescence) is a physical imaging technology, which allows researchers to explore the elemental composition of a study object;

Applied to pictoric image, such a painting or a fresco, or to a document, such miniated manuscripts, allows us to understand the elemental composition of pigments.

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

User may easily visualize the XRF image with few clicks.

GO BACK

43 of 96

43

Kα Pb

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

The XRF (X-Ray Fluorescence) is a physical imaging technology, which allows researchers to explore the elemental composition of a study object;

Applied to pictoric image, such a painting or a fresco, or to a document, such miniated manuscripts, allows us to understand the elemental composition of pigments.

The XRF Analyser service is designed to allow researchers to perform real-time in-browser elaborations and visualization of XRF raw data.

User may easily visualize the XRF image with few clicks.

GO BACK

44 of 96

44

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

45 of 96

45

02-03 May 2022 AI@INFN

AIRES-CH

Neural Network Architectures

1D Branch

MORE DETAILS

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

46 of 96

46

1D: v5_CustomMultInputs

Input Layer (full hist)
Input Layer with normalization (bands)
MaxPool1D (2)
Generate 2-Grams
Conv1D (kernel_size = 3)
Concatenate
BatchNorm

Dense
Dropout
Output
ResBlock (reduce=True)
ResBlock (reduce=False)

.

Input

Conv1D

Input

Conv1D

Input

Conv1D

Generate 2-Grams

Conv1D

ResBlock

Concatenate

.

Input

MaxPool1D

ResBlock

Dense

Dropout

Output

Input

Conv1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Dense

Dropout

Dense

Dropout

Conv1D

ResBlock

2 482 731 parameters

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

47 of 96

47

1D: v5_CustomMultInputs

Input Layer (full hist)
Input Layer with normalization (bands)
MaxPool1D (2)
Generate 2-Grams
Conv1D (kernel_size = 3)
Concatenate
BatchNorm

Dense
Dropout
Output
Conv1D (kernel_size = 1)

Conv1D

BatchNorm

Conv1D

BatchNorm

Conv1D

BatchNorm

Block Input

ResBlock

Conv1D

BatchNorm

Conv1D

BatchNorm

Conv1D

BatchNorm

Block Input

Conv1D

BatchNorm

Residual connections prevents Vanishing Gradient problem

Projection residual block

Identity residual block

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

48 of 96

48

1D: v5_CustomMultInputs

Input Layer (full hist)
Input Layer with normalization (bands)
MaxPool1D (2)
Generate 2-Grams
Conv1D (kernel_size = 3)
Concatenate
BatchNorm

Dense
Dropout
Output
Conv1D (kernel_size = 1)

Conv1D

BatchNorm

Conv1D

BatchNorm

Conv1D

BatchNorm

Block Input

ResBlock

Conv1D

BatchNorm

Projection residual block

Most simplistic explanation would be that 1x1 convolution leads to dimension reductionality. For example, an image of 200 x 200 with 50 features on convolution with 20 filters of 1x1 would result in size of 200 x 200 x 20.

A 1x1xU convolution filter convolved across a V-channel image emulates a UxV matrix multiplied by each V-channel pixel, which is the same as running a single-layer neural network across every pixel of your input as if each pixel were an example vector in a training set.

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

49 of 96

49

1D: v5_CustomMultInputs

Input Layer (full hist)
Input Layer with normalization (bands)
MaxPool1D (2)
Generate 2-Grams
Conv1D (kernel_size = 3)
Concatenate
BatchNorm

Dense
Dropout
Output
Conv1D (kernel_size = 1)

Conv1D

BatchNorm

Conv1D

BatchNorm

Conv1D

BatchNorm

Block Input

ResBlock

Conv1D

BatchNorm

Projection residual block

Most simplistic explanation would be that 1x1 convolution leads to dimension reductionality. For example, an image of 200 x 200 with 50 features on convolution with 20 filters of 1x1 would result in size of 200 x 200 x 20.

A 1x1xU convolution filter convolved across a V-channel image emulates a UxV matrix multiplied by each V-channel pixel, which is the same as running a single-layer neural network across every pixel of your input as if each pixel were an example vector in a training set.

Reduce size

(in filters’ number direction)

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

50 of 96

50

1D: v5_CustomMultInputs

Input Layer (full hist)
Input Layer with normalization (bands)
MaxPool1D (2)
Generate 2-Grams
Conv1D (kernel_size = 3)
Concatenate
BatchNorm

Dense
Dropout
Output
Conv1D (kernel_size = 1)

ResBlock

Conv1D

BatchNorm

Conv1D

BatchNorm

Conv1D

BatchNorm

Block Input

Identity residual block

Most simplistic explanation would be that 1x1 convolution leads to dimension reductionality. For example, an image of 200 x 200 with 50 features on convolution with 20 filters of 1x1 would result in size of 200 x 200 x 20.

A 1x1xU convolution filter convolved across a V-channel image emulates a UxV matrix multiplied by each V-channel pixel, which is the same as running a single-layer neural network across every pixel of your input as if each pixel were an example vector in a training set.

Doesn’t reduce size

(in filters’ number direction)

= Pure identity

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

51 of 96

51

1D: v5_CustomMultInputs

RESULTS

.

Input

Conv1D

Input

Conv1D

Input

Conv1D

Generate 2-Grams

Conv1D

ResBlock

Concatenate

.

Input

MaxPool1D

ResBlock

Dense

Dropout

Output

Input

Conv1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Dense

Dropout

Dense

Dropout

Conv1D

ResBlock

SCORES

The higher, the better:

SSIM mean on 8 test images for model v5_CustomMultInputs test images: 0.38844215869903564

MS-SSIM mean on 8 test images for model v5_CustomMultInputs test images: 0.6803240776062012

PSNR mean on 8 test images for model v5_CustomMultInputs test images: 20.103897094726562

The lower, the better:

BCE mean on 8 test images for model v5_CustomMultInputs test images: 0.6364541140695413

MSE mean on 8 test images for model v5_CustomMultInputs test images: 0.013798782990003625

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

52 of 96

52

1D: v5_CustomMultInputs

RESULTS

.

Input

Conv1D

Input

Conv1D

Input

Conv1D

Generate 2-Grams

Conv1D

ResBlock

Concatenate

.

Input

MaxPool1D

ResBlock

Dense

Dropout

Output

Input

Conv1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Dense

Dropout

Dense

Dropout

Conv1D

ResBlock

GOOD

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

53 of 96

53

1D: v5_CustomMultInputs

RESULTS

.

Input

Conv1D

Input

Conv1D

Input

Conv1D

Generate 2-Grams

Conv1D

ResBlock

Concatenate

.

Input

MaxPool1D

ResBlock

Dense

Dropout

Output

Input

Conv1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Dense

Dropout

Dense

Dropout

Conv1D

ResBlock

GOOD

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

54 of 96

54

1D: v5_CustomMultInputs

RESULTS

.

Input

Conv1D

Input

Conv1D

Input

Conv1D

Generate 2-Grams

Conv1D

ResBlock

Concatenate

.

Input

MaxPool1D

ResBlock

Dense

Dropout

Output

Input

Conv1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Dense

Dropout

Dense

Dropout

Conv1D

ResBlock

NO GOOD

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

55 of 96

55

1D: v5_CustomMultInputs

RESULTS

.

Input

Conv1D

Input

Conv1D

Input

Conv1D

Generate 2-Grams

Conv1D

ResBlock

Concatenate

.

Input

MaxPool1D

ResBlock

Dense

Dropout

Output

Input

Conv1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Conv1D

MaxPool1D

BatchNorm

Dense

Dropout

Dense

Dropout

Conv1D

ResBlock

NO GOOD

Pentagram drawn in the back page

XRF sees behind outermost pictorial layer
too many layers are difficult to recolor

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

56 of 96

56

1D: v6_FractalNet

Input Layer (full hist)
MaxPool1D (2)
Fractal Block
Conv1D (kernel_size = 3)
BatchNorm

Dense
Dropout
Output
Internal block
Flatten

2 117 195 parameters

Input

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Flatten

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

Conv1D

BatchNorm

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

57 of 96

57

1D: v6_FractalNet

RESULTS

Input

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Flatten

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

SCORES

The higher, the better:

SSIM mean on 8 test images for model v6_FractalNet test images: 0.37207430601119995

MS-SSIM mean on 8 test images for model v6_FractalNet test images: 0.6766429543495178

PSNR mean on 8 test images for model v6_FractalNet test images: 20.075578689575195

The lower, the better:

BCE mean on 8 test images for model v6_FractalNet test images: 0.6330596869811416

MSE mean on 8 test images for model v6_FractalNet test images: 0.014069222259422531

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

58 of 96

58

1D: v6_FractalNet

RESULTS

Input

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Flatten

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

GOOD

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

59 of 96

59

1D: v6_FractalNet

RESULTS

Input

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Flatten

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

GOOD

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

60 of 96

60

1D: v6_FractalNet

RESULTS

Input

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Flatten

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

Add

InternalBlock

NO GOOD

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

61 of 96

61

1D: v7_DilatedWaveNet

Input

Causal

Cinv1D

MaxPool1D

Dilated

Conv1D

BatchNorm

Multiply

Conv1D

Add

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Add

GlobalAveragePooling1D

Input Layer
CausalConv (kernel_size = 3)
MaxPool1D (2x2)
WaveNet Block
Conv1D (3)
DilatedConv1D (3, dil_rate = )
BatchNorm
Multiply
Add
GlobalAveragePooling1D
Sigmoid Activation
Tanh Activation
Dense
Dropout
Output

1 809 183 parameters

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

62 of 96

62

1D: v7_DilatedWaveNet

RESULTS

SCORES

The higher, the better:

SSIM mean on 8 test images for model v7_WaveNet test images: 0.3563966155052185

MS-SSIM mean on 8 test images for model v7_WaveNet test images: 0.6734365820884705

PSNR mean on 8 test images for model v7_WaveNet test images: 19.979719161987305

The lower, the better:

BCE mean on 8 test images for model v7_WaveNet test images: 0.6287169173359871

MSE mean on 8 test images for model v7_WaveNet test images: 0.014554956532083451

Input

Causal

Cinv1D

MaxPool1D

Dilated

Conv1D

BatchNorm

Multiply

Conv1D

Add

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Add

GlobalAveragePooling1D

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

63 of 96

63

1D: v7_DilatedWaveNet

RESULTS

GOOD

Input

Causal

Cinv1D

MaxPool1D

Dilated

Conv1D

BatchNorm

Multiply

Conv1D

Add

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Add

GlobalAveragePooling1D

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

64 of 96

64

1D: v7_DilatedWaveNet

RESULTS

GOOD

Input

Causal

Cinv1D

MaxPool1D

Dilated

Conv1D

BatchNorm

Multiply

Conv1D

Add

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Add

GlobalAveragePooling1D

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

65 of 96

65

1D: v7_DilatedWaveNet

RESULTS

NO GOOD

Input

Causal

Cinv1D

MaxPool1D

Dilated

Conv1D

BatchNorm

Multiply

Conv1D

Add

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Add

GlobalAveragePooling1D

the CHNet AIRES-CH project - Alessandro Bombini

02-03 May 2022 AI@INFN

GO BACK

66 of 96

66

02-03 May 2022 AI@INFN

AIRES-CH

Neural Network Architectures

2D Branch

MORE DETAILS

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

67 of 96

67

2D: VGG UNet

Input Layer
Conv2D (3x3)
MaxPool2D (2x2)
Dilated Residual Block
Conv2DT (2x2)
Concatenate+Conv2D (3x3)
Output Layer

15 246 659 parameters

Encoder

VGG-19

Decoder

VGG-19^T

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

68 of 96

68

2D: VGG UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

69 of 96

69

2D: VGG UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

70 of 96

70

2D: VGG UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

71 of 96

71

2D: VGG UNet

RESULTS

MS-SSIM most relevant metric

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

72 of 96

72

2D: VGG UNet

RESULTS

COMPLETE FAILURE

too many layers (?)

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

73 of 96

73

2D: VGG UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

74 of 96

74

2D: VGG UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

75 of 96

75

2D: VGG UNet

RESULTS

NOT SO BAD

why?

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

76 of 96

76

2D: Dilated Residual UNet

Input Layer
Conv2D (3x3)
MaxPool2D (2x2)
Dilated Residual Block
Conv2DT (2x2)
Concatenate+Conv2D (3x3)
Output Layer

2 179 779 parameters

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

77 of 96

77

2D: Dilated Residual UNet

Dilated Residual Block

Block Input
Conv2D (1x1)
Dilated Conv2D (3x3)
BatchNorm
Add

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

78 of 96

78

2D: Dilated Residual UNet

Dilated Residual Block

Block Input
Conv2D (1x1)
Dilated Conv2D (3x3)
BatchNorm
Add

Conv2D

Dilated Conv2D

dilated convolutions are used to increase the receptive field of the higher layers, compensating for the reduction in receptive field induced by removing subsampling.

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

79 of 96

79

2D: Dilated Residual UNet

Dilated Residual Block

Block Input
Conv2D (1x1)
Dilated Conv2D (3x3)
BatchNorm
Add

Conv2D

Dilated Conv2D

Skip connection

Residual connections prevents Vanishing Gradient problem

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

80 of 96

80

2D: Dilated Residual UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

81 of 96

81

2D: Dilated Residual UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

82 of 96

82

2D: Dilated Residual UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

83 of 96

83

2D: Dilated Residual UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

84 of 96

84

2D: Dilated Residual UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

85 of 96

85

2D: Dilated Residual UNet

RESULTS

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

86 of 96

86

2D: Dilated Residual UNet

RESULTS

Here 2D outperforms 1D!

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

GO BACK

87 of 96

87

02-03 May 2022 AI@INFN

AIRES-CH

Conclusions

Analysis of Results

the CHNet AIRES-CH project - Alessandro Bombini

88 of 96

88

02-03 May 2022 AI@INFN

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

XRF

2D UNet-like

1D CNN

Refiner

Recolored Image

Learns from millions of pixels’ histograms
Learns how to associate RGB to peaks distributions
Loses spatial correlations

Learns from few images
Learns how to associate RGB to regions AND peak distributions
Learns spatial correlations

Learns from few images but millions of pixels’ histograms
Learns how to associate RGB to regions AND peak distributions
Learns spatial correlations
The refiner networks learns how to properly merge the two

Up to now, the two branches were developed and trained on a dataset formed by XRF images of various pictorial artworks (e.g.~paintings, drawings, illuminated manuscripts) of different geographical origins and historical periods, and the goal of this project is to show that the goal of inferring an RGB image from an XRF image is feasible.

The DNN is furnished in alpha version in the XRF web app.

the CHNet AIRES-CH project - Alessandro Bombini

89 of 96

89

Calcium

Iron

Example: Ottavio Leoni (late XVII sec. - early XVIII sec.)

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

90 of 96

90

Calcium

Iron

oxide

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

Example: Ottavio Leoni (late XVII sec. - early XVIII sec.)

91 of 96

91

Calcium

Iron

Meudon

white

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

Example: Ottavio Leoni (late XVII sec. - early XVIII sec.)

92 of 96

92

Ottavio Leoni, “Cardinale Antonio Maria Sauli”, 1621.

by courtesy of Accademia “La Colombaria”

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

Example: Ottavio Leoni (late XVII sec. - early XVIII sec.)

93 of 96

93

Van der Weyden, “Lamentation of Christ”, 1460

by courtesy of Galleria degli Uffizi

Ca

Pb

Fe

Hg

Another example: Van der Weyden

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

94 of 96

94

Van der Weyden, “Lamentation of Christ”, 1460

by courtesy of Galleria degli Uffizi

Ca

Pb

Fe

Hg

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

Another example: Van der Weyden

95 of 96

95

Van der Weyden, “Lamentation of Christ”, 1460

by courtesy of Galleria degli Uffizi

Ca

Pb

Fe

Hg

02-03 May 2022 AI@INFN

the CHNet AIRES-CH project - Alessandro Bombini

Another example: Van der Weyden

96 of 96

THE END

Alessandro Bombini

email: bombini@fi.infn.it

	Binary Cross-Entropy	Mean Squared Error
v5_CustomMultInputs	0.636	0.0138
v6_FractalNet	0.633	0.0141
v7_WaveNet	0.629	0.0146

	SSIM	MS-SSIM	PSNR
v5_CustomMultInputs	0.388	0.680	20.104
v6_FractalNet	0.372	0.677	20.076
v7_WaveNet	0.356	0.673	19.980