1 of 42

Running C-ESM-EP on

scylla at CECI-Cerfacs

Laure Coquart

April 2022

Updated April 2024

https://docs.google.com/presentation/d/13374UlbQl287yd3Z-vs4HNhFd9IsATMSLxqAWjzn2eE/edit#slide=id.p

1

2 of 42

Simulation monthly data needed to create

the C-ESM-EP Atlas

- Amon: tas, pr, hfss, hfls, psl, uas, vas, tas, tauu, tauv, hurs, rsutcs, rsut, rlut, rlutcs,rsdt, rsus, rsds, rsdscs, ua, va, ta, hur, hus, zg (grille gr)

- Omon: tos, wfo, sos, zos, mlotst, thetao, so (grille gn)

- SImon: sic (from siconc), sithick, sivol (grille gn)

2

See the Atlas at the address: http://cerfacs.fr/giec6/C-ESM-EP/index_cesmep.php

3 of 42

C-ESM-EP/climaf : python environment (in setenv_C-ESM-EP.sh)

3

New version : C-ESM-EP spirit_ClimafV3 and climaf V3.1 : use cesmep_v3

Install with mamba install (in miniforge3/bin/mamba) install : six, netcdf4, xarray, joblib, matplotlib, scipy, cdms2, cdutil

Creation of the cesmep_env3 virtual environment in python of mamba :

python3 -m venv --system-site-packages /data/scratch/globc/coquart/cesmep_env3

source /data/scratch/globc/coquart/cesmep_env3/bin/activate

pip install --upgrade pip

Old version : C-ESM-EP v2 and climaf 1.2.13

Use cesmep1.0 virtual environment (installed by I. D’Ast)

4 of 42

Copy C-ESM-EP and climaf

located in /data/scratch/globc/coquart on your account

cd /data/scratch/globc/$USER

cp -rf /data/scratch/globc/coquart/C-ESM-EP C-ESM-EP

cp -r /data/scratch/globc/coquart/climaf climaf

All the observations used in the C-ESM-EP are located in a common directory :

/archive/globc/datamg/globc_obs/CESMEP_DATA/ReferenceDatasets

Monthly ERA5 reanalysis CMORised used in the C-ESM-EP are located in common directory :

/archive/globc/datamg/globc_obs/ESMVALTOOL_DATA/cmorized_data/Tier3/ERA5

4

5 of 42

Define the different climaf CACHE in your .bashrc

In /data/scratch/globc/$USER/climaf :

mkdir -p CACHE ; mkdir -p REMOTE_CACHE

Clean the CACHE times to times

In your .bashrc, add the following lines and replace “coquart” by your account name :

export CLIMAF_CACHE=/data/scratch/globc/coquart/climaf/CACHE

export CLIMAF_REMOTE_CACHE=/data/scratch/globc/coquart/climaf/REMOTE_CACHE

export TMPDIR=${CLIMAF_CACHE}

export TEMP=${CLIMAF_CACHE}

export TMP=${CLIMAF_CACHE}

5

6 of 42

Files to add in climaf/climaf/projects

cd /data/scratch/globc/$USER/climaf/climaf/projects
Copy cmip6cerfacs.py into myprojectcerfacs.py
Replace CMIP6CERFACS by your project name in the file myprojectcerfacs.py. Be aware that the facets can be different.
Specify where are located your simulations:

Add the path to the simulations of your project rootproject
Add the pathfilesproject = rootproject+pattern of simulation files
Add another path root2 and another pathfiles2 if necessary
Add this pathfiles to the url keyword in your project dataloc

There will be one root path and one pathfiles for each set of simulations of your project

Add your “myprojectcerfacs” in _ _ init_ _.py

6

7 of 42

To verify/adapt in climaf/climaf

Verify the path of the observations files in projects/ref_climatos_and_ts.py.

The directory ReferenceDatasets is given by IPSL.

Verify the path and the pattern of the ERA5 data and files in projects/ref_era5cerfacs.py.

For data to derive or 3D data level to extract, see derived_variables repository

Boundary definitions for plots defined in plot/plot_params.py

7

8 of 42

C-ESM-EP

In /data/scratch/globc/$USER/C-ESM-EP

standard_comparison is the C-ESM-EP reference repository

with all the diagnostics (components) running at IPSL

standard_comparison_CMIP6CNRMESM2_scenario_ssp585-2015

is the repository that you must adapt to your project and your simulations :

cp -rf standard_comparison_CMIP6CNRMESM2_scenario_ssp585-2015

standard_comparison_my_own_project

8

9 of 42

C-ESM-EP

The definition of the reanalysis ERA5 done in climaf/climaf/projects, is used in the file :

/data/scratch/globc/$USER/C-ESM-EP/custom_obs_dict.py

For the last version (spirit_ClimafV3 + climafV3.1) the file custom_obs_dict_LC_1979-2020.py is located on the Cerfacs Redmine :

https://inle.cerfacs.fr/projects/esmvaltool/wiki/_C-ESM-EP_spirit_ClimafV3_and_climaf_V31_on_coquart_account_25032024_

9

10 of 42

C-ESM-EP

The file

/data/scratch/globc/$USER/C-ESM-EP/cesmep_modules/reference/reference.py

contains the definition of the observations and reanalyses used to compare to the simulation results :

This file is present in CM_atlas/__init__.py and in NEMO_atlas_V2

For the last version (spirit_ClimafV3 + climafV3.1) the file reference.py is located on the Cerfacs Redmine :

https://inle.cerfacs.fr/projects/esmvaltool/wiki/_C-ESM-EP_spirit_ClimafV3_and_climaf_V31_on_coquart_account_25032024_

10

11 of 42

File to modify in C-ESM-EP

Add your project in the file

C-ESM-EP/share/cesmep_modules/NEMO_atlas_v2/plot_NEMO_atlas.py :

# -- Sea Ice Plots

def plot_sic_climato_with_ref(variable, model, ref, season, proj, add_product_in_title=True,

safe_mode=True, custom_plot_params={}, do_cfile=True):

# -- Get the datasets of the model and the ref

wmodel = model.copy()

wmodel['variable'] = variable

if 'table' not in wmodel:

if wmodel['project'] in ['CMIP5']:

wmodel.update(dict(variable=variable, table='OImon'))

if wmodel['project'] in ['CMIP6','CMIP6CERFACS', 'CERFACS_EPOC']:

wmodel.update(dict(variable=variable, table='SImon'))

# wmodel.update(dict(variable=variable, table='OImon'))

wref = ref.copy()

if wref['project'] in ['CMIP6','CMIP6CERFACS', 'CERFACS_EPOC']:

wref.update(dict(variable=variable, table='SImon'))

else:

wref.update(dict(variable=variable, table='OImon'))

11

12 of 42

File to modify in C-ESM-EP

cd /data/scratch/globc/$USER/C-ESM-EP/

In locations.py :

Search for “atCerfacs”

Adapt /data/scratch/globc/dcom/CMIP6_TOOLS/C-ESM-EP/CESMEP_climaf_cache

Adapt /data/scratch/globc/dcom/CMIP6_TOOLS/C-ESM-EP/CESMEP_html

Replace http://cerfacs.fr/giec6/C-ESM-EP/CESMEP_html/ or not with

root_url_to_cesmep_outputs = 'file:///data/scratch/globc/yourname/C-ESM-EP/CESMEP_html/'

12

13 of 42

File to modify in C-ESM-EP for old version (v2.0+v1.2.13)

cd /data/scratch/globc/$USER/C-ESM-EP/

In setenv_C-ESM-EP.sh :

Search for “/data/scratch/globc/”

Modify if [[ -d "/data/scratch/globc/dcom/CMIP6_TOOLS/C-ESM-EP" ]] ; then

Modify export CLIMAF=/data/scratch/globc/dcom/CMIP6_TOOLS/climaf

Modify export cesmep_modules=/data/scratch/globc/dcom/CMIP6_TOOLS/C-ESM-EP/share/cesmep_modules

13

14 of 42

File to modify in C-ESM-EP for new version (spirit_ClimafV3+V3.1)

cd /data/scratch/globc/$USER/C-ESM-EP/

In setenv_C-ESM-EP.sh :

Search for “/data/scratch/globc/”

Replace if [[ -d "/data/scratch/globc/dcom/CMIP6_TOOLS/C-ESM-EP" ]] ; then

By if [[ -d "/data/scratch/globc/$USER/C-ESM-EP" ]] ; then

Replace

# CDAT

source /data/softs/python2/venvs/cesmep1.0/bin/activate

export HDF5_DISABLE_VERSION_CHECK=1

export UVCDAT_ANONYMOUS_LOG=False

By source /data/scratch/globc/coquart/cesmep_env3/bin/activate

14

15 of 42

In setenv_C-ESM-EP.sh :

Replace export CLIMAF=/data/scratch/globc/dcom/CMIP6_TOOLS/climaf

By export CLIMAF=/data/scratch/globc/$USER/climaf

Replace export cesmep_modules=/data/scratch/globc/dcom/CMIP6_TOOLS/C-ESM-EP/share/cesmep_modules

By export cesmep_modules=/data/scratch/globc/$USER/C-ESM-EP/share/cesmep_modules

Replace export CLIMAF_CACHE=/data/scratch/globc/dcom/CMIP6_TOOLS/C-ESM-EP/climafcache_${component}

By export CLIMAF_CACHE=/data/scratch/globc/$USER/C-ESM-EP/climafcache_${component}

15

16 of 42

File to modify in C-ESM-EP for new version (spirit_ClimafV3+V3.1)

cd /data/scratch/globc/$USER/C-ESM-EP/

In main_C-ESM-EP.py :

Search for “frontpage” , comment the lines :

# -- Add link to main frontpage

# DEACTIVATE atCerfacs

#climaf_doc_url = 'https://climaf.readthedocs.io/en/master/'

# -- Replace url to CliMAF documentation with url to C-ESM-EP frontpage

#index = index.replace(climaf_doc_url, cesmep_frontpage)

# -- Replace CliMAF documentation with C-ESM-EP frontpage of comparison COMPARISON

#index = index.replace('CliMAF documentation',

# 'Back to C-ESM-EP frontpage of comparison: '+comparison)

16

17 of 42

File to modify in C-ESM-EP for new version (spirit_ClimafV3+V3.1)

cd /data/scratch/globc/$USER/C-ESM-EP/

In run_C-ESM-EP.py :

Search for “atCerfacs”, replace the initial lines by :

if component == "NEMO_depthlevels":

cmd = 'set -x ; cd ' + submitdir + ' ; export comparison=' + comparison + \

' ; export component=' + component + ' ; export cesmep_frontpage=' + frontpage_address +\

' ; export CESMEP_CLIMAF_CACHE=' + cesmep_climaf_cache + \

' ; sbatch --job-name=CESMEP --partition=prod --nodes=1 --ntasks-per-node=15 ' + \

' --output=cesmep.o --error=cesmep.e -w gsa5 ../' + job_script

else:

cmd = 'set -x ; cd ' + submitdir + ' ; export comparison=' + comparison + \

' ; export component=' + component + ' ; export cesmep_frontpage=' + frontpage_address +\

' ; export CESMEP_CLIMAF_CACHE=' + cesmep_climaf_cache + \

' ; sbatch --job-name=CESMEP --partition=prod --nodes=1 --ntasks-per-node=5 ' + \

' --output=cesmep.o --error=cesmep.e -w gsa5 ../' + job_script

17

18 of 42

File to modify in C-ESM-EP

cd /data/scratch/globc/$USER/C-ESM-EP/standard_comparison_my_own_project

In job_C-ESM-EP.sh :

Search for “atCerfacs”

Replace

if [[ -d "/data/scratch/globc" ]] ; then

CLIMAF_CACHE=/data/scratch/globc/dcom/CMIP6_TOOLS/C-ESM-EP/climafcache_${component}

By

if [[ -d "/data/scratch/globc" ]] ; then

CLIMAF_CACHE=/data/scratch/globc/$USER/C-ESM-EP/climafcache_${component}

18

19 of 42

Files to modify in C-ESM-EP

cd /data/scratch/globc/$USER/C-ESM-EP/

In standard_comparison_my_own_project/datasets_setup.py :

Search for “atCerfacs”

Replace the definition of the simulations by your simulations. The pattern in the dict()

object is linked to what is defined for your project in climaf in the file:

/data/scratch/globc/$USER/climaf/climaf/projects/myprojectcerfacs.py

If you want to compare your results to observations and reanalysis, keep:

reference = 'default'

else if you want to compare to another simulation define it:

reference=dict()

The experiment must correspond to the name of the files of the simulation

19

20 of 42

Files to modify in C-ESM-EP

cd /data/scratch/globc/$USER/C-ESM-EP/standard_comparison_my_own_project

In the component Atmosphere_StdPressLev

Add from custom_obs_dict import custom_obs_dict

Add your project (see CMIP6CERFACS) in params_Atmosphere_StdPressLev.py

The name of your project is defined in climaf in cerfacscmip6.py

+ safe_mode = False

20

21 of 42

Files to modify in C-ESM-EP

cd /data/scratch/globc/$USER/C-ESM-EP/standard_comparison_my_own_project

In the component Atmosphere_Surface

Add from custom_obs_dict import custom_obs_dict

Add your project (see CMIP6CERFACS) in params_Atmosphere_Surface.py.

The name of your project is defined in climaf in cerfacscmip6.py

+ safe_mode = False

21

22 of 42

Files to modify in C-ESM-EP

cd /data/scratch/globc/$USER/C-ESM-EP/standard_comparison_my_own_project

In the component Atmosphere_zonmean

Add from custom_obs_dict import custom_obs_dict

Add your project (see CMIP6CERFACS) in params_Atmosphere_zonmean.py

The name of your project is defined in climaf in cerfacscmip6.py

+ safe_mode = False

22

23 of 42

Files to modify in C-ESM-EP

cd /data/scratch/globc/$USER/C-ESM-EP/standard_comparison_my_own_project

In the component ENSO

Add from custom_obs_dict import custom_obs_dict

Add your project (see CMIP6CERFACS) in params_MainTimeSeries.py

The name of your project is defined in climaf in cerfacscmip6.py

+ safe_mode = False

23

24 of 42

Files to modify in C-ESM-EP

cd /data/scratch/globc/$USER/C-ESM-EP/standard_comparison_my_own_project

In the component MainTimeSeries

Add from custom_obs_dict import custom_obs_dict

Add your project (see CMIP6CERFACS) in params_MainTimeSeries.py

The name of your project is defined in climaf in cerfacscmip6.py

+ safe_mode = False

24

25 of 42

Files to modify in C-ESM-EP

cd /data/scratch/globc/$USER/C-ESM-EP/standard_comparison_my_own_project

In the component NEMO_main

Add from custom_obs_dict import custom_obs_dict

Add your project (see CMIP6CERFACS) in params_NEMO_main.py

The name of your project is defined in climaf in cerfacscmip6.py

+ safe_mode = False

25

26 of 42

Files to modify in C-ESM-EP

cd /data/scratch/globc/$USER/C-ESM-EP/standard_comparison_my_own_project

In the component NEMO_depthlevels

Add from custom_obs_dict import custom_obs_dict

Add your project (see CMIP6CERFACS) in params_NEMO_depthlevels.py

The name of your project is defined in climaf in cerfacscmip6.py

+ set_mode = False

26

27 of 42

Running C-ESM-EP : principle

27

cd /data/scratch/globc/$USER/C-ESM-EP

python run_C-ESM-EP.py standard_comparison_my_own_project component1,component2,component3

This example runs 3 diagnostics (components) in parallel on scylla

run-C-ESM-EP.py ⇒ job-C-ESM-EP.sh

⇒ python main-C-ESM-EP.py - -comparison comparaison - -component composante[i]

28 of 42

Running C-ESM-EP for your components

28

cd /data/scratch/globc/$USER/C-ESM-EP

python run_C-ESM-EP.py standard_comparison_my_own_project component1,component2,component3 [...]

In our case, components are :

MainTimeSeries ; Atmosphere_Surface ; Atmsophere_StdPressLev ; Atmosphere_zonmean ; NEMO_main ; NEMO_depthlevels ; ENSO

contained in the repository :

Standard_comparison_my_own_project

The html files are created in the CESMEP.html repository (defined in locations.py)

29 of 42

Running C-ESM-EP

29

Some repositories climafcache_component are created. Do not forget to remove them, except if you want to add more years to your diagnostics
To modify the queue(s) of submission used by default on scylla, edit the file : run-C-ESM-EP.py and search for atCerfacs
To modify the period over which the diagnostics are done, modify the file : datasets_setup.py
To modify a simulation path and pattern, modify the file :

climaf/climaf/projects/myprojectcerfacs.py

30 of 42

Interesting website links or presentations

30

31 of 42

Atlas Atmosphere_StdPressLev (REF=ERA5) ; CNRM-ESM2-1_ssp585

32 of 42

Atlas Atmosphere_StdPressLev (REF=historical 1984-2014) ; CNRM-ESM2-1_ssp585

33 of 42

Atlas Atmosphere_Surface (REF=ERA5) ; CNRM-ESM2-1_ssp585

34 of 42

Atlas Atmosphere_Surface (REF=historical 1984-2014) ; CNRM-ESM2-1_ssp585

35 of 42

Atlas Atmosphere_zonmean (REF=ERA5) ; CNRM-ESM2-1_ssp585

36 of 42

Atlas Atmosphere_zonmean (REF=historical 1984-2014) ; CNRM-ESM2-1_ssp585

37 of 42

Atlas ENSO (REF=OBS) ; CNRM-ESM2-1_ssp585

38 of 42

Atlas Main Time Series ; CNRM-ESM2-1_ssp585

39 of 42

Atlas Main Time Series (REF=historical 1984-2015) ; CNRM-ESM2-1_ssp585

40 of 42

Atlas NEMO_main (REF=OBS) ; CNRM-ESM2-1_ssp585

41 of 42

Atlas NEMO_main (REF=historical 1984-2014) ; CNRM-ESM2-1_ssp585

42 of 42

Atlas NEMO_depthlevels (REF=OBS)

SSP126 simulations with CNRM-ESM2-1 not SSP585