1 of 61

Enhancing the reliability of deep learning algorithms to improve the observability of French rooftop photovoltaic (PV) installations

CIFRE n°2020/0685

PhD thesis defense – Gabriel Kasmi

5^th April 2024

2 of 61

Context and motivation

3 of 61

Not all PVs are measured equally

PhD thesis defense of Gabriel Kasmi

3

05.04.2024

Photovoltaic energy

No production records

Poor knowledge of the fleet

Image: RTE

Real time measurements

4 of 61

PV grows quickly

Rooftop PV observability:

Ability to estimate the rooftop PV power production with good accuracy.

Two steps:

Construct a PV registry
Estimate PV power production

PhD thesis defense of Gabriel Kasmi

4

05.04.2024

5 of 61

Remote sensing of PV installations

Global need for PV registries

Existing works: deep learning on overhead imagery

PhD thesis defense of Gabriel Kasmi

5

05.04.2024

Image: DeepSolar database (Yu et al., 2018)

6 of 61

Remote sensing of PV installations

Mapping French installations with DeepSolar?

We have our own data requirements,

Sensitivity to distribution shifts.

Deep learning lacks reliability.

PhD thesis defense of Gabriel Kasmi

6

05.04.2024

7 of 61

Remote sensing of PV installations

Framework to improve the reliability of deep learning algorithms:

Monitorable: evaluate the accuracy of the registry,

Auditable: assess the relevance of the model’s decision process,

Robust: ensure that the model is robust to image perturbations.

Construct a registry following these principles and assess its relevance for improving rooftop PV observability.

PhD thesis defense of Gabriel Kasmi

7

05.04.2024

8 of 61

Scientific question

PhD thesis defense of Gabriel Kasmi

8

05.04.2024

Is deep learning-based remote sensing on orthoimagery a suitable method for constructing a nationwide registry of rooftop photovoltaic (PV) installations intended to improve the observability of PV power production in France?

9 of 61

Outline

PhD thesis defense of Gabriel Kasmi

9

05.04.2024

I. Available data I. 1. Geographical information systems (GIS) and existing PV data I. 2. PV registry: data requirements and starting point	10
II. Monitoring with the downstream task accuracy (DTA)	16
III. Auditing the model’s decision process
IV. Assessing the impact on rooftop PV observability	54
Conclusion	66

10 of 61

I. Available data

10

11 of 61

I. 1. GIS and existing PV data

12 of 61

BDORTHO from IGN

Covers France.

Ground sampling distance: 20cm/px.

PhD thesis defense of Gabriel Kasmi

12

05.04.2024

13 of 61

BDTOPO from IGN

PhD thesis defense of Gabriel Kasmi

13

05.04.2024

Shoebox model.

Records the buildings’ locations as geo-localized polygons

14 of 61

The Registre national d’installations

RNI: provided by RTE.

Even internal data is not sufficient for PV power estimation.

PhD thesis defense of Gabriel Kasmi

14

05.04.2024

Excerpt of the RNI restricted to PV systems

15 of 61

BDPV and BDAPPV

Asso BDPV: maintains a rooftop PV database

~28,000 systems’ characteristics,

~4,000 PV yield time series

Basis for BDAPPV (Kasmi et al., 2023a).

PhD thesis defense of Gabriel Kasmi

15

05.04.2024

Number of PV systems registered in BDPV compared to the population of PV systems smaller than 36kWp

16 of 61

I. 2. PV registry: data requirements and starting point

17 of 61

Registry: data requirements

PhD thesis defense of Gabriel Kasmi

17

05.04.2024

Comprehensive

Disaggregated

Technical characteristics: tilt, azimuth and installed capacity

RNI

RTE

Our

registry

BDPV

18 of 61

Building the registry: starting point

PhD thesis defense of Gabriel Kasmi

18

05.04.2024

19 of 61

II. Monitoring: the downstream task accuracy (DTA)

20 of 61

Monitoring the registry

First pilar for reliability.

No labelled data outside the training dataset.

Unsupervised model evaluation method (Zhang et al., 2008).

PhD thesis defense of Gabriel Kasmi

20

05.04.2024

21 of 61

Downstream task accuracy

PhD thesis defense of Gabriel Kasmi

21

05.04.2024

Source: Kasmi et al. (2022)

22 of 61

Visualizing the registry

PhD thesis defense of Gabriel Kasmi

22

05.04.2024

23 of 61

Aggregating detections

PhD thesis defense of Gabriel Kasmi

23

05.04.2024

City-wise aggregated installed capacity [kWp]

24 of 61

Comparing with the RNI

PhD thesis defense of Gabriel Kasmi

24

05.04.2024

Estimation

RNI

25 of 61

Comparing with the RNI

PhD thesis defense of Gabriel Kasmi

25

05.04.2024

Normalized difference between DeepPVMapper and the RNI

Overestimation of the installed capacity

Underestimation of the installed capacity

26 of 61

Comparing with the RNI

PhD thesis defense of Gabriel Kasmi

26

05.04.2024

	DeepPVMapper	RNI
Installed capacity [kWp]	203	80
Number of systems [-]	27	18

27 of 61

Identifying outliers

PhD thesis defense of Gabriel Kasmi

27

05.04.2024

28 of 61

Wrap-up: monitoring

How can we control the registry’s accuracy at scale?

Detailed information on rooftop PV systems is missing.

Existing data sources can help us monitoring the accuracy of the registry.

Our proposal, the DTA, (Kasmi et al., 2022):

Unveils failure cases,
Quantifies the accuracy drop observed in the literature (e.g., De Jong et al., 2020).

PhD thesis defense of Gabriel Kasmi

28

05.04.2024

29 of 61

III. Auditing the model’s decision process

30 of 61

Feature attribution

PhD thesis defense of Gabriel Kasmi

30

05.04.2024

PhD thesis defense of Gabriel Kasmi

30

GradCAM (Selvaraju et al., 2017)

31 of 61

Understanding detections

PhD thesis defense of Gabriel Kasmi

31

05.04.2024

False positives: focuses on something that resembles a PV panel.

32 of 61

Understanding detections

PhD thesis defense of Gabriel Kasmi

32

05.04.2024

False negatives: behaves as if there were not any panel on the image.

33 of 61

A PV panel seen from different scales

PhD thesis defense of Gabriel Kasmi

33

05.04.2024

34 of 61

The wavelet transform of an image

The (dyadic) wavelet decomposition (Mallat, 1989) is a natural way to decompose an image into scales:

PhD thesis defense of Gabriel Kasmi

34

05.04.2024

35 of 61

From the CAM to the Wavelet-CAM

Why not highlighting important wavelet regions instead of image regions?

PhD thesis defense of Gabriel Kasmi

35

05.04.2024

Importance of image regions in the model’s prediction

Importance of wavelet coefficients in the model’s prediction

36 of 61

The wavelet scale attribution method

PhD thesis defense of Gabriel Kasmi

36

05.04.2024

37 of 61

The wavelet scale attribution method

PhD thesis defense of Gabriel Kasmi

37

05.04.2024

Adapted from Kasmi et al. (2023b)

Global sensitivity analysis

(based on Fel et al. 2021)

Black-box model

0.7

0.4

…

0.9

Predicted probabilities

38 of 61

The wavelet scale attribution method

PhD thesis defense of Gabriel Kasmi

38

05.04.2024

Adapted from Kasmi et al. (2023b)

0.7

0.4

…

0.9

Black-box model

Predicted probabilities

39 of 61

Scale disentanglement

PhD thesis defense of Gabriel Kasmi

39

05.04.2024

Scale 1-2px

Importance: 10%

Scale 2-4px

Importance: 27%

Scale 4-8px

Importance: 62%

40 of 61

Understanding false detections

Decomposing the prediction into scales sheds light on false postives.

In Kasmi et al. (2023c), we showed that the disruption of specific scales caused by varying acquisition conditions explains false negatives.

PhD thesis defense of Gabriel Kasmi

40

05.04.2024

41 of 61

Wrap-up: auditability

How can we ensure that the model sees PV panels ?

Traditional explainability methods fail to assess what models see on images,

The WCAM (Kasmi et al. 2023b), highlights which scales contribute to the model’s decision.

Our model indeed relies on relevant scales, but these scales are sometimes disrupted by the varying acquisition conditions (Kasmi et al. 2023c).

PhD thesis defense of Gabriel Kasmi

41

05.04.2024

42 of 61

IV. Assessing the impact on rooftop PV observability

43 of 61

Rooftop PV observability

Observability: ability to derive accurate estimations of the PV power production.

Registry data as an input to PVWatts (Dobos et al. 2014)

PVWatts inputs:

Tilt, azimuth angles and installed capacity
Solar irradiance
Temperature

PhD thesis defense of Gabriel Kasmi

43

05.04.2024

44 of 61

Relevance of the PV registry

Is our registry relevant for improving rooftop PV observability?

Two conditions:

Accuracy: enables an accurate estimations of the rooftop PV power production,

Scalability: the PV power estimation error does not increase as the number of systems increases.

PhD thesis defense of Gabriel Kasmi

44

05.04.2024

45 of 61

Assessment of the accuracy

Ground truth: 900 curated measurements from BDPV:

Oracle: best possible accuracy with the conversion model.

PhD thesis defense of Gabriel Kasmi

45

05.04.2024

46 of 61

Assessment of the scalability

Behavior of the estimation error as the number of installations increases

PhD thesis defense of Gabriel Kasmi

46

05.04.2024

47 of 61

Results: accuracy

We can accurately estimate the individual PV power production using the information provided by the registry and a vanilla conversion model:

PhD thesis defense of Gabriel Kasmi

47

05.04.2024

Table: RMSE [W] and (pRMSE [%]) of the PV power estimation

Case	Mean error [W] (%)
DeepPVMapper	332.57 (10.10)
Oracle	281.53 (8.36)

48 of 61

Results: accuracy

We can accurately estimate the individual PV power production using the information provided by the registry and a vanilla conversion model:

PhD thesis defense of Gabriel Kasmi

48

05.04.2024

Table: RMSE [W] and (pRMSE [%]) of the PV power estimation

Case	Mean error [W] (%)
DeepPVMapper	332.57 (10.10)
Oracle	281.53 (8.36)

49 of 61

Results: scalability

The estimation error decreases as the number of installations increases.

PhD thesis defense of Gabriel Kasmi

49

05.04.2024

50 of 61

Wrap-up: PV observability

How to integrate the registry’s data for improving PV observability?

A conversion model (PVWatts) is required,

Rooftop PV is observable.

Our registry is relevant for improving rooftop PV observability.

PhD thesis defense of Gabriel Kasmi

50

05.04.2024

51 of 61

Conclusion

52 of 61

Conclusion

20% of the PV installed capacity poorly known.

We need to improve our knowledge of the rooftop PV fleet with comprehensive and up-to-date registries.

Deep learning is a promising solution to acquire such information.

Current algorithms cannot be straightfowardly applied as they are not reliable enough.

PhD thesis defense of Gabriel Kasmi

52

05.04.2024

53 of 61

Conclusion

We introduced a method that improves the reliability of deep learning models. The method rests on three pillars:

Pilar #1. Monitoring. DTA, to compare the registry with the RNI,

Pilar #2. Auditing. WCAM, to assess what scales the model sees,

Pilar #3. Robustness. DeepPVMapper and study of the effects of varying acquisition conditions.

We demonstrated that our reliable registry can be used with a simple conversion model to improve rooftop PV observability.

PhD thesis defense of Gabriel Kasmi

53

05.04.2024

54 of 61

Conclusion

We introduced a method that improves the reliability of deep learning models. The method rests on three pillars:

Pilar #1. Monitoring. DTA, to compare the registry with the RNI,

Pilar #2. Auditing. WCAM, to assess what scales the model sees,

Pilar #3. Robustness. DeepPVMapper and study of the effects of varying acquisition conditions.

We demonstrated that our reliable registry can be used with a simple conversion model to improve rooftop PV observability.

PhD thesis defense of Gabriel Kasmi

54

05.04.2024

55 of 61

Conclusion

We introduced a method that improves the reliability of deep learning models. The method rests on three pillars:

Pilar #1. Monitoring. DTA, to compare the registry with the RNI,

Pilar #2. Auditing. WCAM, to assess what scales the model sees,

Pilar #3. Robustness. DeepPVMapper and study of the effects of varying acquisition conditions.

We demonstrated that our reliable registry can be used with a simple conversion model to improve rooftop PV observability.

PhD thesis defense of Gabriel Kasmi

55

05.04.2024

56 of 61

Conclusion

We introduced a method that improves the reliability of deep learning models. The method rests on three pillars:

Pilar #1. Monitoring. DTA, to compare the registry with the RNI,

Pilar #2. Auditing. WCAM, to assess what scales the model sees,

Pilar #3. Robustness. DeepPVMapper and study of the effects of varying acquisition conditions.

We demonstrated that our reliable registry can be used with a simple conversion model to improve rooftop PV observability.

PhD thesis defense of Gabriel Kasmi

56

05.04.2024

57 of 61

Conclusion

We introduced a method that improves the reliability of deep learning models. The method rests on three pillars:

Pilar #1. Monitoring. DTA, to compare the registry with the RNI,

Pilar #2. Auditing. WCAM, to assess what scales the model sees,

Pilar #3. Robustness. DeepPVMapper and study of the effects of varying acquisition conditions.

We demonstrated that our reliable registry can be used with a simple conversion model to improve rooftop PV observability.

PhD thesis defense of Gabriel Kasmi

57

05.04.2024

58 of 61

Limitations

We need fair comparison with the existing PV power estimation models,

We could consider a more detailed approach and do multiclass classification to distinguish PV technologies or PV from thermal solar systems,

Further work is needed for improving the robustness to geographical variability.

PhD thesis defense of Gabriel Kasmi

58

05.04.2024

59 of 61

Future works

WCAM: first step for assessing what models see. Connections with concept-based approaches (e.g., Fel et al., 2023)?

Detection on RGB images: What about multispectral data and data fusion?

PV power production: taking shadings into account?

Good basis for modeling self-consumption.

PhD thesis defense of Gabriel Kasmi

59

05.04.2024

60 of 61

Future works

Mapping France and reporting the computational cost

PhD thesis defense of Gabriel Kasmi

60

05.04.2024

61 of 61

Thank you for your attention

PhD thesis defense of Gabriel Kasmi

61

05.04.2024

Publications	Source code	Databases
Kasmi, G.,et al. (2023a). A crowdsourced dataset of aerial images with annotated solar photovoltaic arrays and installation metadata. Scientific Data, 10(1), 59.	https://github.com/gabrielkasmi/bdappv	https://zenodo.org/records/7347432
Kasmi, G. et al. (2022). Towards unsupervised assessment with open-source data of the accuracy of deep learning-based distributed PV mapping. In MACLEAN workshop at ECML/PKDD 2022.	https://github.com/gabrielkasmi/deeppvmapper
Kasmi, G., et al. (2023b). Assessment of the Reliablity of a Model's Decision by Generalizing Attribution to the Wavelet Domain. In XAI in Action workshop at NeurIPS 2023.	https://github.com/gabrielkasmi/spectral-attribution
Kasmi, G., et al. (2023c). Can We Reliably Improve the Robustness to Image Acquisition of Remote Sensing of PV Systems?. In Tackling Climate Change with Machine Learning workshop at NeurIPS 2023.
Trémenbert, Y., Kasmi, G., et al. (2023). PyPVRoof: a Python package for extracting the characteristics of rooftop PV installations using remote sensing data. arXiv preprint arXiv:2309.07143.	https://github.com/gabrielkasmi/pypvroof

Images: Stable Diffusion