1 of 33

How to download Data

from ESGF nodes

How to get CMIP5, CMIP6, CORDEX data

Laure Coquart, Christian Pagé & Svenya Chripko

March 2021

https://docs.google.com/presentation/d/15sMnMuZZMsZUTDsXJv2ArmwVhU5iriSEY_deB6VXpfw/edit#slide=id.p

2 of 33

3 of 33

4 of 33

5 of 33

ESGF Infrastructure

Publication of data of different projects:

​

  • CMIP6
  • CMIP5
  • Energy Hexascale Earth System Model
  • Obs4MIPS
  • CORDEX
  • European Space Agency’s Climate Change

Initiative Open Data Portal

6 of 33

EXISTING ESGF NODES AND STATUS OF THE NODES

The list of the different existing nodes can be found at the web address:

https://esgf.llnl.gov/nodes.html

The status (UP or DOWN) of the different existing nodes

is given at the web address:

https://esgf-node.llnl.gov/status/

The first step is to register on one existing node.

The OpenID, the username and the password can then be used on

any node after they have been created.

​

Next, example of registration and search of CMIP6 data on the IPSL node (https://esgf-node.ipsl.upmc.fr/projects/esgf-ipsl/)

​

7 of 33

FIRST STEP : register on one node (below Node@IPSL)

Click on “Create an account”

8 of 33

FIRST STEP : register on one node

Choose your username

Fill your first name and last name

Fill your email

Choose a password

Fill your Institution

Fill your city

Fill your country

Upon submission, an OpenID will be automatically assigned to you:

you will need that OpenID to login, with your username and password

9 of 33

FIRST STEP : register on one node

10 of 33

SECOND STEP : get your OpenID and login

11 of 33

SECOND STEP : login

12 of 33

THIRD STEP : search CMIP6 data after login

Scroll down to low on the page to find the search

13 of 33

THIRD STEP : search CMIP6 data

Click on “CMIP6 link” to search for CMIP6 data

14 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

Search tab

Facets to help refine your search

To avoid confusion add search facets at a time and hit “Search” tab above

15 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

First select the source ID (model)

Then click on the “Search” tab

16 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

First filter “Source ID”: CNRM-CM6-1

17 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

Select “Experiment ID”

18 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

For PAMIP, short runs with 100/200/300 members ; Select “Variant Label” (members)

Max ~ 200 = variables x members

in the Data Cart

19 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

Select the first ten members

then click on “Search” tab

20 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

Filter results

21 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

Select your variables (here 8)

then click on “Search” tab

22 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

Filter results

Now you can select which variables will go into your Data Cart (mon&day)

169

23 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

24 of 33

THIRD STEP : search CMIP6 data and create your Data Cart

Visualize your Data Cart

25 of 33

FOURTH STEP : get your Data Cart with wget script

Click on the “wget script” tab

Select all the data

26 of 33

FOURTH STEP : get your Data Cart with wget script

Save the wget script(s) + chmod 755 wget

27 of 33

SIXTH STEP : download the data using the wget script(s)

28 of 33

SIXTH STEP : download the data using the wget script(s)

29 of 33

SIXTH STEP : verify the data that were in your Data Cart

The wget script(s) verify that the data are correctly downloaded

but not if some data are missing from your search

​

It may have been a problem on the node(s) during download,

verify again the download using ./wget.sh -H

​

​

There should have been 2 x 10 x 8 = 160 datasets in the Data Cart

There were only 139 datasets (see slide 23)

​

Verify which data are missing using a shell script and

download again missing data on the ESGF node(s) (see next 2 slides)

30 of 33

SIXTH STEP : verify the data that were in your Data Cart

chmod 755 script_vars_and_members_PAMIP.sh

​

./script_vars_and_members_PAMIP.sh > log 2>&1

31 of 33

Search for PAMIP variable missing and next members

Search tab

If the node where are the data is DOWN or if some data are missing on the node

select “Show all the replicates” or a node in the facet “Data Nodes” on the left to find your missing Data

32 of 33

PROBLEMS ? NEED HELP ?

Send an email to

esgf-user@lists.llnl.gov

with the information below

33 of 33

Interesting web site links

Video Tutorial ESGF Nodes : https://www.youtube.com/watch?v=PTtWfUni0sA

Video on CMIP6 : https://www.youtube.com/watch?v=WdRiYPJLt4o