2 of 18

NIRD IPCC node

NIRD is the (Norwegian) National e-Infrastructure for Research Data. It is owned and operated by UNINETT Sigma2.

3 of 18

Work with ESMValTool on Nird

Nird contains complete CMIP5 and CMIP6 datasets for NorESM, and a subset of CMIP5 and CMIP6 datasets for other models. This makes it attractive to run ESMValTool on Nird. Options include:

Run ESMValTool on the dedicated ipcc.nird.sigma2.no node

Mainly used for CMORization of NorESM output, can be used for other purposes when idle
Requires specific login permission to access

Run ESMValTool through the Nird Toolkit system

Requires access to a Nird Toolkit group, and jupyterlab/jupyterhub application preconfigured with ESMValTool
Access to data across multiple storage volumes is complicated (contact Nird admin)

Install ESMValTool in your own storage volume using the conda system�WARNING: Login nodes are not designed for heavy processing jobs!

4 of 18

ESMValTool on ipcc.nird node

https://noresm-docs.readthedocs.io/en/noresm2/diagnostics/esmvaltool.html#run-esmvaltool-on-nird-service-node

Log in on the ipcc node: ssh -l <username> ipcc.nird.sigma2.no�- For workshop, suggest to use personal directory under /scratch/<username>�
Source conda, do one of the following

Execute source for conda (every time)�module load Miniconda3
Add the line above to your .bashrc or .profile file (one time fix)
First time use: run conda init to initialize conda environments�

Load ESMValTool on Nird (every login): conda activate /diagnostics/esmvaltool/2.8.0�
Copy config file and recipe to where you want to execute esmvaltool�/projects/NS9560K/users/tomast/esmvaltool/noresmvaltool/esmvaltool/config/config-user.yml�/projects/NS9560K/users/tomast/esmvaltool/noresmvaltool/esmvaltool/tested_recipes/�or�git clone https://github.com/NorESMhub/noresmvaltool.git

5 of 18

Install ESMValTool on your own storage volume

Install release version of ESMValTool https://docs.esmvaltool.org/en/latest/quickstart/installation.html #mamba-conda-installation �mamba create --name esmvaltool esmvaltool

Install development version from source�https://docs.esmvaltool.org/en/latest/quickstart/installation.html#install-from-source �git clone https://github.com/ESMValGroup/ESMValTool�mamba env create --name esmvaltool --file environment.yml�conda activate esmvaltool�pip install --editable '.[develop]'

6 of 18

Test installation

After activating the esmvaltool environment, test the installation by running

esmvaltool version

Expected output:� ESMValCore: 2.8.0� ESMValTool: 2.8.0

7 of 18

Get help

Try the following

esmvaltool --help

esmvaltool config --help

esmvaltool data --help

esmvaltool recipes --help

esmvaltool run --help

8 of 18

User config file

Where are input data files located?
What is the directory structure of the input data?
Where should output be placed?
What should ESMValTool consist of?

Output file types
Should intermediate files be preserved?
Level of detail for logs
Profile the diagnostics for efficiency (Python only)

Run tasks in parallel?
Use custom config-developer file?

9 of 18

User config file: CMIP and obsdata on Nird

CMIP data stored using the DKRZ directory structure.

NS9034K : NorESM archive for CMIP5 and CMIP6

/projects/NS9034K/CMIP5/{output1 , output2}/� /projects/NS9034K/CMIP6/

NS9560K-datalake : (mount from /nird/datalake/NS9560K/� CMIP products from various ESMs

/projects/NS9560K-datalake/ESGF/cmip5/{output , output1}/� /projects/NS9560K-datalake/ESGF/CMIP6/

Observational data (for some recipes) are stored in

/projects/NS9560K-datalake/obsdata/{Tier1 , Tier2 , Tier3}/

10 of 18

User config file: options, [...]=default

output_dir: /projects/NS9560K/www/diagnostics/esmvaltool/$USER

Available for access from web browser: http://ns9560k.web.sigma2.no/diagnostics/esmvaltool/

auxiliary_data_dir: /projects/NS9560K-datalake/ESGF/auxiliary_data

e.g. mask files and shapefiles for data selection

search_esgf: { [never], when_missing, always }

Select if missing data should be downloaded from ESGF to download_dir

download_dir: /scratch/$USER/ESGF
max_parallel_tasks: { [null], 1, 2, 3, … } # null = use all available CPUs
log_level: { debug, [info], warning, error }
exit_on_warning: { true, [false] }
output_file_type: { [png], pdf, ps, eps, epsi }
remove_preproc_dir: { [true], false }
compress_netcdf: { true, [false] }
save_intermediate_cubes: { true, [false] }
config_developer_file: { [null], custom “config-developer.yml”
profile_diagnostic: { true, [false] } # profiler for python diagnostics

11 of 18

Developer’s configuration file

c onfig-developer.yml : � Project- and machine-dependent directory and file name definitions of the� input and output data.

12 of 18

Available standard recipes

Standard recipes bundled with the esmvaltool installation: list and copy

esmvaltool recipes list� esmvaltool recipes get <standard_recipe.yml>

List of standard recipes tested on NIRD with ESMValTool 2.8.0:

https://docs.google.com/spreadsheets/d/1V7epqQgzPZXdLk_VBBlRH0Zq6yg_6gbRl-kgQ9UgDHE/edit?usp=sharing

(location may change to a more permanent site after workshop)

13 of 18

ESMValTool run files

To run an ESMValTool diagnostics, it is necessary (and usually sufficient) to provide the config and recipe files

config-user.yml : General settings for ESMValTool
recipe_python.yml : How to execute the diagnostics

A diagnostics run is executed from the command line

esmvaltool run --config_file=config-user.yml examples/recipe_python.yml

Optional: ESMValTool looks for a default user config file in

$HOME/.esmvaltool/config-user.yml

Copy your config file to this location to avoid using the --config_file flag for every run.

14 of 18

Run a standard recipe

List available recipes:�esmvaltool recipes list

Show content of a recipe:�esmvaltool recipes show examples/recipe_python.yml

Copy a recipe to local directory (optional for standard recipes):�esmvaltool recipes get examples/recipe_python.yml

Run esmvaltool with recipe:�esmvaltool run --config_file=config-user.yml ./recipe_python.yml

See output: http://ns9560k.web.sigma2.no/diagnostics/esmvaltool/

15 of 18

ESMValTool output

An ESMValTool run creates a directory with the following output subdirectories

run/ : Run logs, copy of recipe, summary of resource usage
preproc/ : NetCDF output from preprocessing runs (optional)

This is removed when config option remove_preproc_dir: true is set

plots/ : Graphical output
work/ : Any output that is not a plot, e.g. NetCDF files after diagnostic run

Most standard recipes also create an index.html file in the root directory that shows a summary and usually the graphical output of the recipe.

16 of 18

ESMValTool output: logs

Main logs:

run/main_log.txt - short log
run/main_log_debug.txt - full debug log
run/<diag_name>/log.txt - log for each diagnostic task

17 of 18

Problems when running esmvaltool recipes

NIRD does not contain a full set of CMIP5/6 and observational datasets.

Running an untested standard recipe will probably fail on Nird at the first attempt

Missing observational datasets

Solution1: Download from ESGF if available, using --search_esgf=when_missing
Solution2: Add dataset, CMORize if needed (contact esmvaltool-on-nird discussion group)

Missing model datasets

Solution1: Try to run with --skip_nonexistent=True flag
Solution1: Download from ESGF if available, using --search_esgf=when_missing
Solution2: Replace missing dataset with something available (e.g. NorESM data)

Missing auxiliary data, e.g. shapefile or mask

Solution1: Copy from /project/NS9560K-datalake/ESGF/auxiliary_data/
Solution2: Add auxiliary data in directory according to your config file

Missing support for diagnostic script (mainly problem with Julia)

18 of 18

Resources: Further reading and getting help

NorESM documentation in ReadTheDocs: ESMValTool diagnostics

From GitHub: NorESMhub/noresmvaltool

GitHub NorESMhub team: esmvaltool-on-nird discussion group