1 of 9

JASMIN, Hybrid Cloud and bringing the Compute to the Data for ESGF

Update on CMIP-6 Data Status, Delivery and Analysis Capabilities

AMS 2020

Philip Kershaw

2 of 9

CMIP6 on JASMIN Headline Figures

Primary data archive

Replica data

Data selected / prioritised on the basis of the IPCC WG1 data priority list (2019).

MOHC: mostly complete some high frequency data to be published next year, tbc.

NERC: ongoing data expected throughout 2021

Data published under CMIP6/HighResMIP and PRIMAVERA

Subset of data to be made available to Copernicus Climate Data Store

3 of 9

Evolution to regional clusters aggregating data + co-locating it alongside processing capability

Single data centre

Federated data repositories: ESGF (2008)

Big Data driving changes in architecture

Private/Community Cloud + HPC Hybrid e.g. JASMIN (2012)

Public Cloud

Data Analysis Platforms

Co-located compute and data
Analysis Ready Data (ARD)

Evolution != replacement

Evolution == augmentation + replacement

4 of 9

Federation enables scaling and resilience

Node

ESGF Nodes at sites around the world deliver data and associated services

5 of 9

Federation and Cloud can augment and complement one another

Centralised Cloud Services for federation

Node

On-premise nodes for big data serving – research community labs, private and community clouds, computing centres

Federation glue: search, identity, metrics, monitoring public cloud-hosted

6 of 9

JASMIN – Data-intensive computing and community cloud hybrid

POSIX disk

Object store

S3

OpenStack

JASMIN Cluster-as-a-Service (CaaS)

tape

Datasets straddling storage media for fluid workflows

API

JASMIN Notebook Service

7 of 9

Data-aware subsetting and regridding service for Climate Data

Analyse datasets and make an inventory of fixes

Applies fixes to datasets before applying subsetting/regridding operations

C3S 34e Project

Credit: Ag Stephens, CEDA

8 of 9

Ongoing work and next steps

ESGF Future Architecture:� - centralized services on cloud� - Docker/Kubernetes Deployment architecture

Collaboration on CMIP data on object store: �- Pangeo community, AWS ESGF node, DKRZ, …

9 of 9

Acknowledgements + Contact Details

This work has been carried out through IS-ENES3, a project funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 824084
Subsetting and regridding service developed for C3S 34e project has been funded by the Copernicus Climate Change Service. ECMWF implements this Service on behalf of the European Commission.

Philip.Kershaw@stfc.ac.uk

@PhilipJKershaw