Machine Learning

for Cultural Heritage

Alessandro Bombini, INFN-CHNet Firenze & ICSC

XX Seminar on Software

for Nuclear, Subnuclear and Applied Physics,

Porto Conte, June, 5th, 2023

Physics

Nuclear Imaging

Raw Data

of

Pictorial Artworks

Digital restoration

Possible application:

2

Before ML:

Nuclear Vision

for Cultural Heritage

Before ML

​

​

Stratigraphy of Pictorial Artworks

​

​

wood, canvas, etc.

organic glues, etc.

Pb white, Ca white, etc.

graphite

4

Before ML: Stratigraphy of Pictorial Artworks

Physical Imaging techniques

wood, canvas, etc.

organic glues, etc.

Pb white, Ca white, etc.

graphite

5

Before ML: Stratigraphy of Pictorial Artworks

Signals

6

WHAT IS X-RAY FLUORESCENCE?

7

Before ML: Stratigraphy of Pictorial Artworks

Creating the XRF datacube

8

Before ML: Stratigraphy of Pictorial Artworks

Creating the XRF datacube

motor movement

9

Fe

WHAT IS X-RAY FLUORESCENCE?

10

Fe

11

Why X-Ray Fluorescence?

​

​

• It provides fast, sensitive, multi-elemental non-invasive, non-destructive analysis. It is perfect for Cultural Heritage applications;​
• It can be performed with portable apparatus for in-situ analysis (e.g. in museums).​
• Can produce macro-maps [~ O(meter)] (MA-XRF)​
• It is able to detect signal coming from hidden pictorial layers, underneath the outermost one.​

• IT IS AVAILABLE TO US.

12

• It provides fast, sensitive, multi-elemental non-invasive, non-destructive analysis. It is perfect for Cultural Heritage applications;​
• It can be performed with portable apparatus for in-situ analysis (e.g. in museums).​
• Can produce macro-maps [~ O(meter)] (MA-XRF)​
• It is able to detect signal coming from hidden pictorial layers, underneath the outermost one.​

• IT IS AVAILABLE TO US.

INTRINSIC LIMITS OF X-RAY FLUORESCENCE

• Slow pace dataset creation (~O(10) XRF raw data / year)
• Noisy data ( e.g. calibration ADC-to-Energy changes)
• Data engineering needed (e.g. alignment/resizing of RGB pixels to XRF pixels)
• Low detection limits of light elements
• Blind to organic properties of pigments (they are always C-O-H compounds)

INTRINSIC LIMITS OF RGB

• Non absolute color scale
• Data engineering needed (e.g. alignment/resizing of RGB pixels to XRF pixels)
• Color perception is subjective (it is a psychophysical phenomenon)
• Pigments color is not absolute & changes in time (color degradation)

Are we doomed to fail?

(spoiler: no)

14

INTRINSIC LIMITS OF X-RAY FLUORESCENCE

• Slow pace dataset creation (~O(10) XRF raw data / year)
• Noisy data ( e.g. calibration ADC-to-Energy changes)
• Data engineering needed (e.g. alignment/resizing of RGB pixels to XRF pixels)
• Low detection limits of light elements
• Blind to organic properties of pigments (they are always C-O-H compounds)

INTRINSIC LIMITS OF RGB

• Non absolute color scale
• Data engineering needed (e.g. alignment/resizing of RGB pixels to XRF pixels)
• Color perception is subjective (it is a psychophysical phenomenon)
• Pigments color is not absolute & changes in time (color degradation)

Are we doomed to fail?

(spoiler: no)

A NEW HOPE?

clustering MA-XRF raw data

16

A New Hope

clustering MA-XRF raw data

NOTE: NO DEEP LEARNING HERE

MA-XRF RAW DATA

17

A New Hope

clustering MA-XRF raw data

NOTE: NO DEEP LEARNING HERE

MA-XRF RAW DATA

Dimensional reduction

linear

18

A New Hope

clustering MA-XRF raw data

NOTE: NO DEEP LEARNING HERE

MA-XRF RAW DATA

Manifold Learning

highly nonlinear

19

A New Hope

clustering MA-XRF raw data

MA-XRF RAW DATA

Dimensional reduction

linear

20

A New Hope

clustering MA-XRF raw data

MA-XRF RAW DATA

Manifold learning

highly nonlinear

t-distributed Stochastic Neighbour Embedding

21

A New Hope

clustering MA-XRF raw data

MA-XRF RAW DATA

Manifold learning

highly nonlinear

https://pair-code.github.io/understanding-umap/

22

A New Hope

clustering MA-XRF raw data

k-means clustering

https://www.naftaliharris.com/blog/visualizing-k-means-clustering/

23

A New Hope: clustering MA-XRF raw data

​

real case application: detached fresco

Recently, in order to value the sensitivity of the system some tests were carried out with particular attention to the detection of residual material on degraded surfaces.

​

This is particularly relevant for example for the frescoes, in which degradation can be caused by several external agents involving physical, chemical and/or biological factors, like structure displacements, vibrations, temperature changes, humidity, pollution and mildew. This could bring to the discoloration of the upper painted layer of the fresco and, depending on the extent of the damages, it can bring the loss of the legibility of the artwork.

​

​

A mockup was prepared in the laboratory and it is a replica of an Etruscan wall painting from IV century BC from the "Tomba della Quadriga Infernale" (Tomb of the Infernal Quadriga) in Sarteano (Siena)

​

The fresco was obtained following the historical buon fresco technique, consisting of the application of a dry-powder pigment dispersed in water and drawn upon freshly "wet" lime plaster.

24

A New Hope: clustering MA-XRF raw data

​

real case application: detached fresco

In order to simulate the loss of pigment the intonachino was "detached" from the lime plaster support and re-collocated on a linen canvas, following the so-called technique of "strappo"

Therefore, the measurements of the residual pigment were carried out with the MA-XRF scanner both on the detachment and on the plaster background, containing the residual of the pigments.

​

25

A New Hope: clustering MA-XRF raw data

​

real case application: detached fresco

In order to simulate the loss of pigment the intonachino was "detached" from the lime plaster support and re-collocated on a linen canvas, following the so-called technique of "strappo"

Therefore, the measurements of the residual pigment were carried out with the MA-XRF scanner both on the detachment and on the plaster background, containing the residual of the pigments.

​

26

A New Hope: clustering MA-XRF raw data

​

real case application: detached fresco

It seems to be an insurmountable issue.

​

27

A New Hope: clustering MA-XRF raw data

​

real case application: detached fresco

It seems to be an insurmountable issue.

​

UMAP

(latent space viz)

28

A New Hope: clustering MA-XRF raw data

​

real case application: detached fresco

It seems to be an insurmountable issue.

​

clustering in latent space

29

A New Hope: clustering MA-XRF raw data

​

real case application: detached fresco

It is not an insurmountable issue.

​

30

A New Hope: clustering MA-XRF raw data

​

takeaway

• MA-XRF raw data presents many hidden statistical relations we can exploit
• We can either use peaks relations (i.e. 1D properties), or
• We can use spatial segmentation (i.e. 2D properties)

31

Artificial Intelligence for digital REStoration of Cultural Heritage

the AIRES-CH project

What is AIRES-CH?

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

33

What is AIRES-CH?

​

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

1D Branch

34

What is AIRES-CH?

​

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

2D Branch

35

What is AIRES-CH?

​

Artificial Intelligence for digital REStoration of Cultural Heritage (AIRES-CH) aims at building a web-based app for the digital restoration of pictorial artworks through Computer Vision technologies applied to physical imaging raw data.

The goal is to develop a DNN capable of inferring the RGB image from an XRF image (i.e. the 3D tensor [h,w,E]); this will be obtained by a multi-dimensional DNN, capable of exploiting features of 1D and 2D DNN.

Joint branches

36

Training dataset

AIRES-CH

37

Training Dataset

The whole dataset is composed by 62 XRF raw data coming from several XRF analysis on multiple paintings performed both in the LABEC facility in Florence, as well as in situ analysis (the data comes not only from published works, and include some private artworks).

For training the 1D models, only a 50% of the pixels where used (being randomly chosen), giving a training dataset of around 2,059,780 [histogram, RGB] pairs, divided into training, test, and validation set.

For training the 2D models, 45 XRF scans are used, reserving the remaining as 9 for test, and 8 for validation.

The raw data are obtained by three different devices, all developed, built and assembled by CHNet.

The raw data comes from different artwork typologies: multi-layered paintings, drawings without preparatory layers, and illuminated manuscripts, all over different periods and epochs (from middle ages to contemporary art).

It is worth noticing that artworks from different epochs might have been realised using different pigments. Visually similar colour can therefore be associated to completely different XRF spectra depending on the painting’s epoch.

Bombini, A., Anderlini, L., dell’Agnello, L., Giacomini, F., Ruberto, C., Taccetti, F.: The AIRES-CH project: Artificial Intelligence for digital REStoration of Cultural Heritages using physical imaging and multidimensional adversarial neural networks, Accepted for publication on the ICIAP2021 conference proceedings, Springer Lecture Notes in Computer Science vol. 13231

38

Neural Network Architectures

1D & 2D branches

AIRES-CH

39

Neural Network Architectures

1D Branch

2D Branch

We have developed and trained few 1D - DNN models:

1. Dense
2. CNN
3. ResNet-like
4. Inception-like
5. Custom Model
6. FractalNet
7. (Dilated)WaveNet

​

We have developed and trained fewer 2D-DNN models:

• VGG-like
• DilResNet-like

Due to the higher computational costs AND for the (relatively) small dataset size

​

40

Neural Network Architectures: a primer

​

Dense (Multi-layer perceptrons)

Input

We actually can understand it

41

Neural Network Architectures: a primer

​

​

Dense (Multi-layer perceptrons)

Multi-layer Perceptron (a.k.a. first Deep Neural Network)

feed forward

42

Neural Network Architectures: a primer

Training the network

1. Initialise

• Loop:
1. Compute feed-forward prediction:

• compare with real data:

Q: HOW?

A: Through the LOSS function

• update params

43

Neural Network Architectures: a primer

Training the network: backprop

44

Neural Network Architectures: a primer

Convolutional neural networks

If we change the feed-forward process, we change the DNN architecture;

45

Neural Network Results

1D & 2D branches

AIRES-CH

46

Neural Network Architectures

1D Branch

2D Branch

We have developed and trained few 1D - DNN models:

• Dense
• CNN
• ResNet-like
• Inception-like
• Custom Model
• FractalNet
• (Dilated)WaveNet

​

We have developed and trained fewer 2D-DNN models:

• VGG-like
• DilResNet-like

Due to the higher computational costs AND for the (relatively) small dataset size

​

All of them were (moderately) capable of inferring the RGB from the XRF histogram.

• Structural Similarity Index Measure (SSIM)
• Multi-Scale SSIM (MS-SSIM)
• Peak Signal-to-Noise Ratio (PSNR)

We checked the

performances using:

• Binary Cross-Entropy (BCE)
• Mean Squared Error (MSE)

We trained the model

using:

47

Neural Network Architectures

1D Branch

2D Branch

We have developed and trained few 1D - DNN models:

• Dense
• CNN
• ResNet-like
• Inception-like
• Custom Model
• FractalNet
• (Dilated)WaveNet

​

We have developed and trained fewer 2D-DNN models:

• VGG-like
• DilResNet-like

Due to the higher computational costs AND for the (relatively) small dataset size

​

All of them were (moderately) capable of inferring the RGB from the XRF histogram.

• Structural Similarity Index Measure (SSIM)
• Multi-Scale SSIM (MS-SSIM)
• Peak Signal-to-Noise Ratio (PSNR)

We checked the

performances using:

• Binary Cross-Entropy (BCE)
• Mean Squared Error (MSE)

We trained the model

using:

48

Neural Network Architectures

1D Branch

We have developed and trained few 1D - DNN models:

• Dense
• CNN
• ResNet-like
• Inception-like
• Custom Model
• FractalNet
• (Dilated)WaveNet

​

49

Neural Network Architectures

1D Branch

We have developed and trained few 1D - DNN models:

• Dense
• CNN
• ResNet-like
• Inception-like
• Custom Model
• FractalNet
• (Dilated)WaveNet

​

Zhou Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," in IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, April 2004, doi: 10.1109/TIP.2003.819861.

50

Example on validation raw data - multilayered painting

​

​

Good

1D branch

51

Example on validation raw data - multilayered painting

​

​

Good

1D branch

52

Example on validation raw data - illuminated manuscript

​

NOT Good

1D branch

53

Example on validation raw data - illuminated manuscript

​

NOT Good

2D branch

Here 2D outperforms 1D

54

AIRES-CH web app

A cloud-native application for digital restoration of Cultural Heritage using nuclear imaging: AIRES-CH

55

Conclusions & Outlook

• We have shown that each MA-XRF datacube contains a huge amount of information
• we can exploit it in many ways
• We have shown that the goal of inferring an RGB image from an XRF raw data is feasible.
• We have developed both 1D & 2D branches
• We have started a new measurement campaign, jointly with Biblioteca Marucelliana, Firenze, on their drawings, to enlarge and standardise the training dataset.
• We have embedded an alpha version on the network inside our XRF web tool

​

56

Conclusions & Outlook

• We have developed a way to generate a synthetic dataset, to overcome the small dataset issue, by means of our python package ganX - generate artificially new XRF [arXiv:2304.14078], by means of dictionary learning.
• it creates a MA-XRF datacube out of an RGB image + a palette consisting of a set of couples (XRF signal, rgb color)
• We are going to scrape the internet for appropriate RGB images (i.e., Italian Medieval Manuscripts), and we use the appropriate palette to generate a huge, standardised dataset.
• This dataset will be used to train a generative model (such as a Variational Auto-Encoder), which eventually will be fine-tuned to real data, to furthermore refined the synthetic dataset.

​

• We have shown that each MA-XRF datacube contains a huge amount of information
• we can exploit it in many ways
• We have shown that the goal of inferring an RGB image from an XRF raw data is feasible.
• We have developed both 1D & 2D branches
• We have started a new measurement campaign, jointly with Biblioteca Marucelliana, Firenze, on their drawings, to enlarge and standardise the training dataset.
• We have embedded an alpha version on the network inside our XRF web tool

57

Alessandro Bombini

email: bombini@fi.infn.it

Thanks for your attention!

59

Alessandro Bombini

email: bombini@fi.infn.it

Thanks for your attention!

Alessandro Bombini

email: bombini@fi.infn.it

Extra slides

ganX

how to create a synthetic MA-XRF datacube out of RGB

more on…

62

ganX

One of the main issues encountered during AIRES-CH project was the limited size of the training dataset.

​

To address this issue, we starte the

generating artificially new Xrf (ganX)¹

project (to be read as [gan-ex]).

​

ganX is a python library which allow for a synthetic MA-XRF datacube generation out of an RGB and a palette of [RGB color, XRF spectrogram] couples.

​

It performs a dictionary learning, where the coefficient of the linear combination is the similarity in the CIE2000 space between the pixel color and the palette RGB color.

how to create a synthetic MA-XRF datacube out of RGB

ganX logo artwork by Serena Barone

63

ganX

how to create a synthetic MA-XRF datacube out of RGB

64

Deep Neural Network Architectures

1D branch

more on…

65

66

• We have developed and trained few DNN models:
1. Dense
2. CNN
3. ResNet-like
4. Inception-like
5. Custom Model
6. FractalNet
7. (Dilated)WaveNet

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

ResNet50 (2015)

Residual connections address Vanishing Gradient problem

67

• We have developed and trained few DNN models:
• Dense
• CNN
• ResNet-like
• Inception-like
• Custom Model
• FractalNet
• (Dilated)WaveNet

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

Inception (2014)

68

• We have developed and trained few DNN models:
• Dense
• CNN
• ResNet-like
• Inception-like
• Custom Model
• FractalNet
• (Dilated)WaveNet
• All of them were (moderately) capable of inferring the RGB from the XRF histogram.
• Three of them were slightly more performing

1D BRANCH MODELS

the CHNet AIRES-CH project - Alessandro Bombini

69

• We have developed and trained few DNN models:
• Dense
• CNN
• ResNet-like
• Inception-like
• Custom Model
• FractalNet
• (Dilated)WaveNet

1D BRANCH MODELS

Oord, Aaron van den, et al. "Wavenet: A generative model for raw audio." arXiv preprint arXiv:1609.03499 (2016). https://doi.org/10.48550/arXiv.1609.03499

the CHNet AIRES-CH project - Alessandro Bombini

70

the CHNet AIRES-CH project - Alessandro Bombini

71

1D: v7_DilatedWaveNet

1 809 183 parameters

the CHNet AIRES-CH project - Alessandro Bombini

72

1D BRANCH MODELS

• We have developed and trained few DNN models:
• Dense
• CNN
• ResNet-like
• Inception-like
• Custom Model
• FractalNet
• (Dilated)WaveNet

Larsson, G., Maire, M., & Shakhnarovich, G. (2016). Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648. https://doi.org/10.48550/arXiv.1605.07648

the CHNet AIRES-CH project - Alessandro Bombini

73

the CHNet AIRES-CH project - Alessandro Bombini

74

1D: v6_FractalNet

• Input Layer (full hist)
• MaxPool1D (2)
• Fractal Block
• Conv1D (kernel_size = 3)
• BatchNorm

• Dense
• Dropout
• Output
• Internal block
• Flatten

2 117 195 parameters

Input

Input

Dense

Dropout

Dense

Dropout

Dense

Dropout

Output

Flatten

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

Add

InternalBlock

InternalBlock

InternalBlock

InternalBlock

Conv1D

BatchNorm

the CHNet AIRES-CH project - Alessandro Bombini

75

1D BRANCH MODELS

why WaveNet & FractalNet?

Zabihi, M., Rad, A. B., Kiranyaz, S., Särkkä, S., & Gabbouj, M. (2019). 1d convolutional neural network models for sleep arousal detection. arXiv preprint arXiv:1903.01552.

https://doi.org/10.48550/arXiv.1903.01552

Overview of different 1D DNN for a completely different task

• ResNet
• FractalNet
• WaveNet

the CHNet AIRES-CH project - Alessandro Bombini

76

• We have developed and trained few DNN models:
• Dense
• CNN
• ResNet-like
• Inception-like
• Custom Model
• FractalNet
• (Dilated)WaveNet

1D BRANCH MODELS

Tin (Sn) Kα

Potassium (K) Kα

Manganese (Mn) Kα

Titanium (Ti) Kα

the CHNet AIRES-CH project - Alessandro Bombini

77

1D: v5_CustomMultInputs

• Input Layer (full hist)
• Input Layer with normalization (bands)
• MaxPool1D (2)
• Generate 2-Grams
• Conv1D (kernel_size = 3)
• Concatenate
• BatchNorm

• Dense
• Dropout
• Output
• ResBlock (reduce=True)
• ResBlock (reduce=False)

.

.

.

.

.

.

Input

Conv1D

Conv1D

Input

Conv1D

Conv1D

Input

Conv1D

Conv1D

Generate 2-Grams

Conv1D

Conv1D

Conv1D

Conv1D

Conv1D

Conv1D

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

ResBlock

Concatenate

.

.

.

.

.

.

Input

MaxPool1D

Dense

Dropout

Output

Input

Conv1D

BatchNorm

Dense

Dropout

Dense

Dropout

Conv1D

2 482 731 parameters

the CHNet AIRES-CH project - Alessandro Bombini

Deep Neural Network Architectures

2D branch

more on…

78

79

• Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:
• VGG-like
• DilResNet-like

2D BRANCH MODELS

2D: VGG UNet

15 246 659 parameters

the CHNet AIRES-CH project - Alessandro Bombini

80

• Due to the higher computational costs, we have developed and trained 2 DNN UNet-like models:
• VGG-like
• DilResNet-like

2D BRANCH MODELS

2D: Dilated Residual UNet

2 179 779 parameters

the CHNet AIRES-CH project - Alessandro Bombini

Visual Scores

.

more on…

81

82

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

83

dynamic range

Structural Similarity (SSIM):

Zhou Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," in IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, April 2004, doi: 10.1109/TIP.2003.819861.

more on Visual Scores

84

Structural Similarity (SSIM):

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

85

Multiscale Structural Similarity (MS-SSIM):

more on Visual Scores

Z. Wang, E. P. Simoncelli and A. C. Bovik, "Multiscale structural similarity for image quality assessment," The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, 2003, pp. 1398-1402 Vol.2, doi: 10.1109/ACSSC.2003.1292216.

the CHNet AIRES-CH project - Alessandro Bombini

86

Peak Signal-to-Noise Ratio (PSNR):

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

87

Peak Signal-to-Noise Ratio (PSNR):

True Image

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

88

Peak Signal-to-Noise Ratio (PSNR):

Signal

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

89

Peak Signal-to-Noise Ratio (PSNR):

Noise

more on Visual Scores

the CHNet AIRES-CH project - Alessandro Bombini

90

more on Visual Scores

Damon M. Chandler, "Seven Challenges in Image Quality Assessment: Past, Present, and Future Research", International Scholarly Research Notices, vol. 2013, Article ID 905685, 53 pages, 2013. https://doi.org/10.1155/2013/905685

the CHNet AIRES-CH project - Alessandro Bombini

UMAP

.

more on…

91

more on…

UMAP

https://arxiv.org/pdf/1802.03426.pdf

92

more on… UMAP

Part I

https://arxiv.org/pdf/1802.03426.pdf

93

more on… UMAP

Part II

https://arxiv.org/pdf/1802.03426.pdf

94

Alessandro Bombini

email: bombini@fi.infn.it

Extra slides