CleF - Finding CMIP data
Scott Wales & Paola Petrelli
CLEX CMS
Irmer, Reiner. Shelves Laden with Books Awaiting Reshelving at the State Library of Queensland, 1991 (CC-BY)
cws_help@nci.org.au
CMIP data at NCI
NCI has CMIP data in several projects:
cws_help@nci.org.au
CleF
CleF is a tool for searching through these spaces, finding what datasets are available at NCI and where their files are
/g/data
Master
Postgresql
Database
Clef
Postgresql
Database
CleF
Tools
Filesystem Crawler
Ingests Metadata
Sqlalchemy-based library
cws_help@nci.org.au
Clex Conda Environment
CleF is available in the Conda environment
module use /g/data/hh5/public/modules�module load conda/analysis3
Available from both Raijin & VDI
Python 3 + a large variety of climate and weather-relevant libraries
Updates are first put in 'conda/analysis3-unstable'
cws_help@nci.org.au
CleF Searches
If files are available locally, CleF returns a list of canonical paths to that dataset at NCI
clef cmip5 --model ACCESS1.3 --experiment historical � --frequency mon --variable tas
/g/data1/rr3/publications/CMIP5/output1/CSIRO-BOM/ACCESS1-0/historical/mon/atmos/Amon/r3i1p1/latest/tas�/g/data1/rr3/publications/CMIP5/output1/CSIRO-BOM/ACCESS1-0/historical/mon/atmos/Amon/r2i1p1/latest/tas�/g/data1/rr3/publications/CMIP5/output1/CSIRO-BOM/ACCESS1-0/historical/mon/atmos/Amon/r1i1p1/latest/tas
By default CleF will use ESGF's API to make sure you get the most recent file versions - if a dataset has been unpublished from ESGF it may not show up
cws_help@nci.org.au
CleF Searches
If files are not available locally, CleF returns ESGF dataset IDs
clef cmip5 --model MIROC5 --experiment rcp85 --frequency day � --realm atmos --ensemble r1i1p1 --variable clt
Available on ESGF but not locally:�cmip5.output1.MIROC.MIROC5.rcp85.day.atmos.day.r1i1p1.v20120710 clt
cws_help@nci.org.au
Requesting Data Downloads
You can request data to be downloaded with '--request'
clef --request cmip5 --model MIROC5 --experiment rcp85� --frequency day --realm atmos --ensemble r1i1p1� --variable clt
Available on ESGF but not locally:�cmip5.output1.MIROC.MIROC5.rcp85.day.atmos.day.r1i1p1.v20120710 clt
Finished writing file: CMIP5_saw562_20190903T131818.txt�Do you want to proceed with request for missing files? (N/Y)
cws_help@nci.org.au
CleF Options
clef [ ACTION ] DATASET [FACETS]
ACTION can be:
DATASET can be:
FACETS are viewable with � 'clef DATASET --help'
cws_help@nci.org.au
Search Facets
You can search using most of the options available on ESGF https://esgf.nci.org.au/search/esgf-nci/
ESGF search terms are called 'Facets' and include (for CMIP5) Institute, Model, Experiment, Realm, Ensemble, CMOR Table, Variable and Frequency
CMIP5 and CMIP6 have different facet names, check what's available with e.g. 'clef cmip6 --help'
cws_help@nci.org.au
More Complex Searches
Find all models that share a list of facets with '--and', e.g. models with both 'tas' and 'clt' variable_ids
clef cmip6 --experiment historical --frequency mon� --member r1i1p1f1 --variable tas --variable clt� --and variable_id
/g/data1b/oi10/replicas/CMIP6/CMIP/BCC/BCC-CSM2-MR/historical/r1i1p1f1/Amon/clt/gn/v20181126/�/g/data1b/oi10/replicas/CMIP6/CMIP/BCC/BCC-CSM2-MR/historical/r1i1p1f1/Amon/tas/gn/v20181126/�/g/data1b/oi10/replicas/CMIP6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/clt/gn/v20181214/�/g/data1b/oi10/replicas/CMIP6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/tas/gn/v20181214/�/g/data1b/oi10/replicas/CMIP6/CMIP/CAMS/CAMS-CSM1-0/historical/r1i1p1f1/Amon/clt/gn/v20190708/�/g/data1b/oi10/replicas/CMIP6/CMIP/CAMS/CAMS-CSM1-0/historical/r1i1p1f1/Amon/tas/gn/v20190708/
cws_help@nci.org.au
CleF development is on Github - https://github.com/coecms/clef
We have some future goals�- CSV output�- CORDEX support
Feature requests are welcome
Thanks to NCI for assistance with the MAS Database
cws_help@nci.org.au