Earth System Grid Federation: a globally distributed data archive for climate data

ESGF Dashboard: http://esgf-ui.cmcc.it

ESGF – Application and Evolution

ESGF Frontend

ESA Climate Change Initiative Open Data Portal

CMIP/CORDEX for Copernicus Climate Data Store

CMIP5 >> Earth System Grid Federation >> CMIP6

ESA Climate Change Initiative Open Data Portal

2 Phases:

1. Leveraged ESGF
1. Quick win with search and download
2. Bespoke search API incompatible with other community standards - OGC CSW
3. THREDDS Data Server couldn’t scale to our needs

​

• Redesigned to address issues
• OpenSearch API replaced ESG Search
• Scalable data service with Kubernetes
• Zarr format cache of netCDF data on object store for performance

C3S 34[a-f] Projects for the CDS

C3S Resilient CMIP and CORDEX Data Access

CDS

Replicate netCDF model data

Replicate netCDF model data

download data

Sub-setting Services for C3S 34e Project

Analyse datasets and make an inventory of fixes

Applies fixes to datasets before applying subsetting/regridding operations

C3S 34e Project

Credit: Ag Stephens, CEDA

https://nbviewer.org/github/roocs/rooki/blob/master/notebooks/demo/demo-rooki-subset-by-point.ipynb

ESA Earth Observation Exploitation Platform Common Architecture (EOEPCA)

• Architectural blueprint for the federation of platforms
• Interlinked with the work on recent OGC testbeds
• CEDA involved with consultancy role
• Processing and chaining of particular interest
• ADES and EMS
• Ability to push customised shrink-wrapped processes to 3^rd party WPS instances
• Innovations with ID management: UMA

ESGF – Application and Evolution

ESGF Frontend

ESA Climate Change Initiative Open Data Portal

CMIP/CORDEX for Copernicus Climate Data Store

CMIP5 >> Earth System Grid Federation >> CMIP6

ESGF Future Architecture

• Container and Container+Kubernetes installs available
• Deployed on AWS (GFDL) and at CEDA

• Major community engagement on use of STAC for ESM data
• Prototype developed by CEDA
• Integration tests
• CoG and MetaGrid futures??

• OpenID Connect / OAuth 2.0 done
• New Authorisation system with Open Policy Agent
• Authentication integrated with C4I in test

Progress and Achievements

ESGF Future Architecture

• Factored out TDS
• Test CMIP6 data caches on object store at CEDA and DKRZ

​

​

• Ported C3S WPS Data Reduction Services for use in ENES CDI
• Used with Climate4Impact
• Reboot of Compute Working Team

• New Metrics system integrated with CMCC

​

​

​

Progress and Achievements

Future Architecture Node – Phase 1

Kubernetes Cluster

Horizontal Pod Auto-scaler

Auto-Scaling

Elastic

POSIX Storage

Access Control

Identity Provider

TDS

Nginx File Serving

ESG Search

V5 esg-publisher

Ingress

Solr

Client App – Search + Data Access

Metrics

Future Architecture Node – Phase 2

Site Deployment(s)

Centralised Deployment(s)

Kubernetes Cluster

Kubernetes Cluster

Horizontal Pod Auto-scaler

Auto-Scaling

Elastic

POSIX Storage

Access Control

Identity Provider

TDS

Nginx File Serving

STAC Search

Elastic Search

Client App - Data Access

Ingress

Metrics

Ingress

Client App – Data Search

Client App – Publishing

STAC API for ESGF

• Implementation of STAC API
• ElasticSearch backend
• Filter extensions to support faceted search
• Fully featured STAC equivalent API to ESG Search
• Simple frontend created to demonstrate its features

IS-ENES3 - Data Analytics using Notebooks/icclim

DestinE and Blueprint Architecture

Destination Earth (DestinE) - major EU initiative:

• “to develop a very high precision digital model of the Earth (a ‘digital twin’) to monitor and predict environmental change and human impact to support sustainable development”

��

​

JASMIN

Cloud Infrastructure

Data Sources

Data Analytics Platform

High Performance Computing

Data production / processing

ESA Digital Twin Earth (DestinE) Precursor - land surface modelling and climate

• Using JULES (Joint UK Land Environment Simulator)
• the land surface component in the Met Office Unified Model

​

• Improvements with Data Assimilation
• LaVEnDAR (The Land Variational Ensemble Data Assimilation fRamework)
• Feed in satellite observations – SIF and SMAP data

​

19

What could be the future impact of climate change on the soil moisture?

​

​

• JULES driven with climate projections from ISIMIP data (Inter-Sectoral Impact Model Intercomparison Project)

20

Make a surrogate AI model to JULES

• Experimented with Machine Learning (ML) techniques
• Goal: a general-purpose algorithm -

�time series of daily weather data 🡺 time series of soil moisture data

​

• Successfully applied XGBoost (eXtreme Gradient Boosting) algorithm.
• trained on up to 1000 grid cells, representative of the various biomes in continental Africa
• Demonstrated to accurately emulate JULES output at other locations
• The credibility of the model is enhanced by its transparency and explainability

​

21

Digital Twin Precursor on JASMIN: HPC for data production, cloud for analysis

Arrangement of data and efficient access

• Data output from models as netCDF format
• Data in files arranged in spatial dimensions one per time step
• But predominate access pattern for analysis of climate data in the project is time series query (grey blocks)

​

Latitude

Longitude

Object Store: Different storage strategies showed radically different performance

​

• We experimented with different storage chunking arrangements
• 20-year dataset of soil moisture

Using Object Store for re-arrangement of data to suite our access patterns

• Using zarr and xarray Python libraries to store and access the data

​

• Chunked data into a series of strips along the time axis

Rechunking of data made possible interactive maps with long time series

Take home message: object store for analysis-ready cache specific to project needs

• Object store can be efficient for access on cloud
• It is essential to orient data storage to suit predominant access patterns
• Good news – re-writing data into different orientations was fast

​

Futures

Acknowledgements + Further Info

@ISENES_RI

@cedanews

@PhilipJKershaw

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement N°824084

IS-ENES3 website

https://is.enes.org/

Contact us at

is-enes@ipsl.fr

Subscribe to the IS-ENES3 H2020 Youtube channel !

ESGF Future Architecture Report: https://doi.org/10.5281/zenodo.3928222