ThoMaS – PACE Hack Week

Juan Ignacio Gossn

Remote Sensing Specialist – Ocean Colour - EUMETSAT

Eumetsat Allee 1, 64295 Darmstadt, Germany

JuanIgnacio.Gossn@eumetsat.int

​

+ Eleni Kalogeraki, Ilaria Cazzaniga, Malcolm Taberner, Ewa Kwiatkowska, Ben Loveday, Hayley Evers-King, David Dessailly, Anna E. Wyndle di Paola, James G. Allen, Dirk Aurin

​

PACE Hack Week

August 2024

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

1

1. What’s ThoMaS? Scope

ThoMaS is a toolkit developed to create matchups of bio-geophysical insitu data with satellite ocean colour products from Sentinel-3 OLCI (S3/OLCI).

in SeaBASS format

Standard products from NASA’s OBPG also supported

Others easily configurable, if netCDF or series of netCDFs

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

2

1. What’s ThoMaS? Scope

After running ThoMaS, you will get:

• Insitu data “transformed” to match satellite (spectral convolution, band-shifting, BRDF…).
• Satellite data (L1B – TOA radiance - or L2 – BOA water reflectance) from EUMETSAT Data Store and NASA OBPG (reprocessed/operational) matching spatially/temporally your insitu.
• Extractions of satellite data centred at lat/lon of insitu of user-defined size (3x3, 5x5..).
• Statistics of extractions following EUMETSAT’s or any user-defined matchup protocol.
• Merging of simultaneous (spatially-temporally) insitu-satellite pairs, temporal interpolation, and statistics of matchups.

​

• Outputs:
• NetCDF 4 files: SatData, minifiles, Extraction Data Base files, In situ Data Base file, Matchup Data Base files.
• CSV: summarizing satellite extraction statistics and matchup statistics.
• PNG: Standardised output plots.

​

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

3

1. Motivation for having ThoMaS

1. Well documented, suited also for a first approach to the matchup exercise for those who are new to the intricacies of the matchup steps.
2. It’s publicly available, free and open to scrutiny: it serves for the purpose of converging to a standard matchup practice.
3. It supports the most commonly used matchup protocols in the OC community.
• e.g. of existing ones: EUMETSAT’s, Bailey & Werdell 2006, Zibordi 2009, Copernicus SVC_VIS
• Versatile: new matchup protocols can be easily added via configuration files.
• It contains an easy syntax to implement quality flags based on simple relations among products.
4. It deals (under some assumptions) with propagation of uncertainties to the performance metrics (using a Monte-Carlo approach).
5. Already supports some of the most commonly used OC satellite missions
• Currently supports Sentinel-3 (standard) L1B, L2, MODIS L2 (standard l2gen), VIIRS L2 (standard l2gen), SeaHawk L2 (standard l2gen), and now PACE (standard l2gen, AOP products).
• Versatile: new types of satellite products can be easily added via configuration files (depending on mission, processor and processing baseline).

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

4

1. Some disclaimers: ThoMaS is just a baby!

1. The policy of garbage-in garbage-out applies to ThoMaS: you must know well your insitu data, the satellite product, and make sure that you are comparing “apples” to “apples”. Also, that your satellite pixels are sufficiently away from land, and rationally choose your extraction size, time difference tolerance, among many others.
2. ThoMaS does not deal with issues regarding spatial and temporal collocation of insitu and satellite data beyond very standard QC (e.g. a maximum time tolerance window, a choice of window size, and a simple temporal interpolation). This means that the problem of spatial and temporal autocorrelation of the Rrs signal (and any other OC product) is not yet dealt within ThoMaS.
3. ThoMaS cannot still compute match-up statistics of a given insitu-satellite set with varying satellite extraction sizes.
4. The uncertainty of the satellite component is only based on the inter-pixel variability (pixel-by-pixel uncertainties in the satellite component are still ignored in ThoMaS).
5. The uncertainties of the BRDF step are not propagated. Morel 2002 and Lee 2011 approaches available.
6. ThoMaS won’t do an A/C of your satellite data!
7. Many other disclaimers (working on many of these ☺), but I hope it still proves useful!
8. Many people use it just for downloading the satellite data matching their in situ and performing the extractions.

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

5

2. Some background: match-ups

• What is a match-up according to chatGPT?

Ocean colour 👀

Of course we have much more to define… and take care of…

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

6

2. Some background: match-ups

Are we comparing 🍎 with 🍎?

🡪 What is the definition of Rrs?

🡪 Are these two compatible “spectrally”? 🡪 convolution/band-shifting

🡪 Are these two compatible “directionally”? 🡪 BRDF correction

BRDF correction:

Spectral convolution

Burggraaff 2020

D’Alimonte et al.

Morel et al. 2002 supported in ThoMaS

Definition of Rrs

OO Web Book, Mobley, Boss & Roesler

Band-shifting (to pair multispectral to multispectral)

Melin & Sclep 2015 supported in ThoMaS

PACE: Spectral reconstruction

Talone, Zibordi, Pitarch 2024, recently added

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

7

2. Some background: match-ups (spectrally matching insitu to satellite)

An issue with the Melin & Sclep 2015 band-shifting method (designed for multi2multi matching)

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

8

2. Some background: match-ups (spectrally matching insitu to satellite)

Tried Talone, Zibordi and Pitarch 2024 method

​

​

This method:

1. Finds the best 3 matches from a set of exhaustive hyperspectral Hydrolight simulations, using a selected set of multispectral bands.
2. Establishes a blending approach in a way that the resulting reconstructed spectrum coincides exactly with the insitu original spectrum at the original multispectral bands and takes the spectral dependence of these best 3 matches in the intermediate “query” bands.

​

​

​

Validated with a concurrent hyperspectral in situ instrument, showing relative differences typically < 5 % between modelled and measured values.

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

9

2. Some background: match-ups (spectrally matching insitu to satellite)

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

10

2. Some background: match-ups

Are we comparing 🍎 with 🍎?

• Are the insitu measurements of sufficient quality?
• Are insitu and satellite measurements temporally-spatially comparable?
• What value (and uncertainty) shall I extract from the satellite data?

Quality of insitu

ThoMaS still does not consider any quality flag to process insitu data…

Define your extraction statistics!

Define your extraction window size!

Define the matchup statistics!

EUMETSAT’s Matchup Protocols

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

11

2. Some background: match-ups: EUMETSAT extraction protocol

EUMETSAT’s Matchup Protocols: extraction of statistics at macropixel level

Set BFOR: 5x5 window

Detection of non-valid pixels

(flagged pixels)

Set BOR: without flagged pixels

Detection of outliers

Set final: without flagged & outlier

Pixels are masked/removed if flagged by any of the following:

​

CLOUD, CLOUD_AMBIGUOUS, CLOUD_MARGIN, INVALID, COSMETIC, SATURATED, SUSPECT, HISOLZEN, HIGHGLINT, SNOW_ICE, AC_FAIL, WHITECAPS, ADJAC, RWNEG_O2, RWNEG_O3, RWNEG_O4, RWNEG_O5, RWNEG_O6, RWNEG_O7, RWNEG_O8

+ product-specific flags e.g. OC4ME_FAIL

Pixel ‘X’ is considered outlier if:

​

|value@X - µ_BOR| < 1.5σ_BOR

​

Macropixel is discarded if:

​

N_BOR < 50% N_BFOR

Central value: median_final

Uncertainty measure (Type B): σ_final

Homogeneity measure: CV_final

Macropixel is discarded if:

​

CV_final(560)>20%

µ 🡪 Mean

σ 🡪 Standard deviation

Window size recommended: 5x5 or 3x3

Tolerable insitu-satellite time difference: 1 hr or 3 hrs

ThoMaS can be run with many other extraction protocols, that you define

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

12

2. Some background: match-ups: Bailey & Werdell protocol

EUMETSAT’s Matchup Protocols: extraction of statistics at macropixel level

Set BFOR: 5x5 window

Detection of non-valid pixels

(flagged pixels)

Set BOR: without flagged pixels

Detection of outliers

Set final: without flagged & outlier

Pixels are masked/removed if flagged by any of the following:

​

ATMFAIL, LAND, HIGLINT, HILT, HISATZEN, STRAYLIGHT, CLDICE, COCCOLITH, HISOLZEN, LOWLW, CHLFAIL, NAVWARN, MAXAERITER, CHLWARN, ATMWARN, SEAICE, NAVFAIL, MODGLINT

+ product-specific flags e.g. OC4ME_FAIL

Pixel ‘X’ is considered outlier if:

​

|value@X - µ_BOR| < 1.5σ_BOR

​

Macropixel is discarded if:

​

N_BOR < 50% N_BFOR

Central value: median_final

Uncertainty measure (Type B): σ_final

Homogeneity measure: CV_final

Macropixel is discarded if:

​

Median[CV(Rrs(410-551));CV(AOT(869))]>15%

µ 🡪 Mean

σ 🡪 Standard deviation

Window size recommended: 5x5 or 3x3

Tolerable insitu-satellite time difference: 1 hr or 3 hrs

ThoMaS can be run with many other extraction protocols, that you define

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

13

2. Some background: match-ups: what protocol to use?

Should we care about what matchup protocol we use?

[Concha et al. 2021]

YES

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

14

2. Some background: match-ups, performance metrics used

Mean Difference 🡪

Mean Absolute Difference 🡪

Mean Percent Difference 🡪

Mean Absolute Percent Difference 🡪

Log-based Mean Absolute Difference 🡪

Following recommendation by Seegers et al. 2018

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

15

2. Some background: match-ups, performance metrics used

Median Difference 🡪

Median Absolute Difference 🡪

Median Percent Difference 🡪

Median Absolute Percent Difference 🡪

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

16

2. Some background: match-ups, performance metrics used

Spectral Angle Mapper 🡪

Chi-squared 🡪

+ linear regression of two types…

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

17

3. Run the code

Example 9: Casablanca Platform (PACE and VIIRS)

PI: Marco Talone and Frederic Melin

[Zibordi et al. 2002]

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

18

3. Run the code

Example 9: Casablanca Platform (PACE and VIIRS)

1. You want to test the performance of OLCI at the AERONET-OC station Casablanca, South West of Spain (Med Sea) during March to July 2024.

​

2. You wish to get matchups between this Casablanca subset and
• NOAA-20/VIIRS and PACE/OCI standard FR L2,
• From the operational OBPG collection
• using the extraction protocol from Bailey and Werdell 2006,
• an extraction window of 3x3,
• an insitu-satellite time difference threshold of 1 hour (3600 seconds).

​

• You are not interested in getting ancillary data from ECMWF for to the insitu data.

​

• You want to apply the Morel et al. 2002 BRDF correction to both satellite and insitu.

​

• You may have several insitu measurements corresponding to one single SatData within the time window that you selected, but you wish to keep only the closest in time with the satellite overpass.

​

• You wish: SatData to be stored at /path/to/Casablanca/SatData

​

• all the other outputs (IDB, minifiles, EDB, MDB, etc.) to be stored at /path/to/Casablanca

In this case, latLonTimeRanges.csv will be generated automatically by ThoMaS (based on your inputted insitu lat-lon and timestamps + your inputted time tolerance in config_file.ini) and stored under path_output

[global]

path_output: /path/to/Casablanca

SetID: Casablanca

​

[workflow]

workflow: insitu, SatData, minifiles, EDB, MDB

​

[AERONETOC]

AERONETOC_pathRaw:/path/to/AERONET_OC_raw

AERONETOC_dateStart: 2024-03-01T00:00:00

AERONETOC_dateEnd: 2024-08-01T00:00:00

AERONETOC_dataQuality: 1.5

AERONETOC_station: Casablanca_Platform

​

​

[insitu]

insitu_input: /path/to/Casablanca/Casablanca_Platform_OCDB.csv

insitu_satelliteTimeToleranceSeconds: 3600

insitu_getAncillary: False

insitu_BRDF: M02

​

[satellite]

satellite_path-to-SatData: /path/to/Casablanca/SatData

satellite_source: NASA_OBPG

satellite_collections: operational

satellite_platforms: NOAA-20, PACE

satellite_resolutions: FR

satellite_BRDF: M02

​

[minifiles]

minifiles_winSize: 3

​

[EDB]

EDB_protocols_L2: Bailey_and_Werdell_2006

EDB_winSizes: 3

​

[MDB]

MDB_time-interpolation: insitu2satellite_NN

MDB_stats_plots: True

MDB_stats_protocol: EUMETSAT_standard_L2

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

19

3. Run the code: command line

If YES.. run by executing this command:

python /path/to/ThoMaS/main.py –cf /path/to/config_file.ini

1. EUMETSAT Data Store credentials obtained and stored?
2. (optional) ECMWF ADS/CDS credentials obtained and stored?
3. ThoMaS code cloned?
4. thomas conda environment set up and activated?
5. Required inputs in place? (config_file.ini, insitu input file?)

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

20

3. Run the code: Try the new GUI ☺ 🡪 git pull origin main

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

21

4. Now a DEMO

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

22

4. What you should get at the MDB step

Essentially:

1. Matchup results in netCDF4

2. Matchup results in CSV

3. Matchup plots in PNG

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

23

4. What you should get at the MDB step

NOAA-20/VIIRS

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

24

4. What you should get at the MDB step

PACE/OCI

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

25

4. What you should get at the MDB step

NOAA-20/VIIRS

PACE/OCI

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

26

4. What you should get at the MDB step

NOAA-20/VIIRS

PACE/OCI

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

27

4. What you should get at the MDB step

NOAA-20/VIIRS

PACE/OCI

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

28

4. What you should get at the MDB step

NOAA-20/VIIRS

PACE/OCI

copernicus.eumetsat.int

ThoMaS - PACE Hack Week

29

5. Pre-requisites

• Apart from that background knowledge…
• Conda: Install the latest Anaconda Python distribution.
• EUMETSAT Data Store:
• Create EO Portal user and get API consumer key and secret.
• Save EO Portal API credentials under ~/.eumdac_credentials.json
• NASA OBPG:
• Create Earthaccess user and get user and password.
• Save user and password under ~/.obpg_credentials.json
• ECMWF: Register to ADS/CDS and get url and key.
• ECMWF: store ADS/CDS url/keys under ~/.ecmwf_api_config.txt

​

​

Dependencies

Conda will take care of this…

copernicus.eumetsat.int

EUM/OPS-COPER/TEM/15/813104, v2, 3 November 2021

30

6. Getting the code

• Git way:

cd ~

mkdir ThoMaS

cd ThoMaS

git clone --depth 1 https://gitlab.eumetsat.int/eumetlab/oceans/ocean-science-studies/ThoMaS .

https://gitlab.eumetsat.int/eumetlab/oceans/ocean-science-studies/ThoMaS

• Direct download:

Recent updates were done on the code

git pull origin main

copernicus.eumetsat.int

EUM/OPS-COPER/TEM/15/813104, v2, 3 November 2021

31

7. Setting the environment

• Once conda and ThoMaS are installed, create the thomas env:

cd ~

cd ThoMaS

conda env create –f environment.yml

conda activate thomas

libmamba is the best choice for those of you who are stuck in the “Solving environment step”

Questions?: JuanIgnacio.Gossn@eumetsat.int

copernicus.eumetsat.int

EUM/OPS-COPER/TEM/15/813104, v2, 3 November 2021

32