1 of 42

Design of coupled Earth System models

​

David Lawrence

CESM Chief Scientist

​

with lots of input from Mariana Vertenstein

​

This material is based upon work supported by the National Center for Atmospheric Research, which is a major facility sponsored by the National Science Foundation under Cooperative Agreement No. 1852977.

2 of 42

Global Earth System Models

  • Earth System (climate) models use physical equations to simulate key fields and processes in the atmosphere, ocean, land, sea-ice, land-ice, …
  • Processes that remain below the grid resolution need to be parameterized
  • Build on our understanding of processes from observations and highly-detailed models (e.g., process models, large eddy simulations)

3 of 42

Structure of a fully coupled Earth System Model

Atmosphere

Ocean

​

Coupler

​

Sea Ice

​

Land

Chemistry

Biogeochemistry

​

Land Ice

​

River

​

Transfer of information between components: energy, water, carbon, etc

4 of 42

Structure of a fully coupled Earth System Model

Atmosphere

Ocean

​

Coupler

​

Sea Ice

​

Land

Chemistry

Biogeochemistry

​

Land Ice

​

River

​

Forcings:

  • Greenhouse gases
  • Anthropogenic aerosols
  • Volcanic eruptions
  • Solar variability
  • Land use change

5 of 42

Community Earth System Model (CESM3)

Atmosphere

(CAM7)

Ocean

(MOM6)

Coupler

(CIME2)

Sea Ice

(CICE6)

Land

(CTSM6

BGC-Crop, FATES)

Chemistry

(CAM7-Chem)

Biogeochemistry

(MARBL)

​

Land Ice

(CISM3)

High-Top Atm

(WACCM7)

Surface Waves

(WaveWatch3)

Rivers

(MOSART, mizuRoute)

6 of 42

Pulling it all together

7 of 42

Community Earth System Model (CESM3)

Atmosphere

(CAM7)

Ocean

(MOM6)

Coupler

(CIME2)

Sea Ice

(CICE6)

Land

(CTSM6

BGC-Crop, FATES)

Chemistry

(CAM7-Chem)

Biogeochemistry

(MARBL)

​

Land Ice

(CISM3)

High-Top Atm

(WACCM7)

Surface Waves

(WaveWatch3)

Rivers

(MOSART, mizuRoute)

Aiming to release

CESM3 towards the end of the year or early next year

8 of 42

Different research questions require a flexible modeling framework

9 of 42

Different research questions require a flexible modeling framework

Internal vs Forced

Winter Precipitation Trend (2010-2060)

Ens. avg.

Wettest

Driest

Large ensembles

climate change uncertainty

+CO2

10 of 42

Different research questions require a flexible modeling framework

Global

km-scale

Regionally- refined

High resolution modeling

Accurate simulation weather phenomena and extremes

11 of 42

Different research questions require a flexible modeling framework

Complexity

12 of 42

CESM flexibly supports a range of Earth System research through a single model code base

Atmosphere

(CAM6)

Ocean

(POP2)

Coupler

(CIME)

Sea Ice

(CICE5)

Land

(CLM5

BGC-Crop)

Chemistry

(CAM6-Chem)

Biogeochemistry

(MARBL)

​

Land Ice

(CISM2)

High-Top Atm

(WACCM6)

Surface Waves

(WaveWatch2)

Rivers

(MOSART)

13 of 42

CESM flexibly supports a range of Earth System research through a single model code base

Data Atmosphere

Ocean

(POP2)

Coupler

(CIME)

Sea Ice

(CICE5)

For example, you can turn on and turn off different components and replace them with a Data model

T, P, Solar Radiation, Longwave Radiation, Humidity, Wind, CO2, etc. from observations (reanalysis)

14 of 42

CESM flexibly supports a range of Earth System research through a single model code base

Data Atmosphere

Ocean

(POP2)

Coupler

(CIME)

Sea Ice

(CICE5)

Biogeochemistry

(MARBL)

​

For example, you can turn on and turn off different components and replace them with a Data model

T, P, Solar Radiation, Longwave Radiation, Humidity, Wind, CO2, etc. from observations (reanalysis)

15 of 42

CESM flexibly supports a range of Earth System research through a single model code base

Data Atmosphere

Coupler

(CIME)

Land

(CLM5

BGC-Crop)

Rivers

(MOSART)

For example, you can turn on and turn off different components and replace them with a Data model

16 of 42

CESM flexibly supports a range of Earth System research through a single model code base

Data Atmosphere

Coupler

(CIME)

Land

(CLM5

BGC-Crop)

  • Land use change
  • Biogeochemistry
  • Crops
  • Prescribed vegetation
  • Ozone damage
  • Methane emissions
  • # soil layers
  • No anthro
  • …

And, within each component model, there are many choices you can make about parameterizations, complexity levels, resolution, etc

17 of 42

CESM flexibly supports a range of Earth System research through a single model code base

Atmosphere

(CAM6)

Ocean

(POP2)

Coupler

(CIME)

Sea Ice

(CICE5)

Land

(CLM5

BGC-Crop)

Chemistry

(CAM6-Chem)

Biogeochemistry

(MARBL)

​

Land Ice

(CISM2)

High-Top Atm

(WACCM6)

Surface Waves

(WaveWatch2)

Rivers

(MOSART)

18 of 42

Coupling infrastructure should provide ability to

  • facilitate collaboration from a wide user community
  • address requirements from multiple modeling systems
  • facilitate ‘hierarchical model development’ (including easy substitution of a ‘data’ component for a prognostic one)
  • easily introduce and use new model grids (including unstructured and refined grids) - online re-gridding is vital!
  • easily add new fields to be exchanged between components

​

​

​

Coupler

(CIME

ESMF/NUOPC, CMEPS, CDEPS)

19 of 42

Coupling infrastructure should provide ability to

Coupler

(CIME

ESMF/NUOPC, CMEPS, CDEPS)

The Common Infrastructure for Modeling the Earth (CIME) is a python-based framework that is shared between CESM and other modeling centers to ensure efficient coordination of model development

20 of 42

What is ESMF/NUOPC?

The Earth System Model Framework (ESMF) is software for building coupled Earth System Models from interoperable model components

  • high-performance capabilities such as run-time grid remapping and optimized parallel communication

​

Within ESMF is the National Unified Operational Prediction Capability (NUOPC) Layer, which simplifies the interoperability of ESMF components:

  • Introduces four generic, extensible components with built-in capabilities
  • Driver, Connector, Model, Mediator

https://earthsystemmodeling.org/

21 of 42

NUOPC

  • Driver: Provides a harness for Models, Mediators, and Connectors, coordinating their initialization and driving them during the application time loop
  • Model: Typically implements a specific physical domain, e.g. atmosphere, ocean, sea ice, waves, etc.
  • Mediator: Used for custom coupling code (flux calculations, averaging, etc.) between multiple Models
  • Connector: Connects pairs of components, e.g. Model to/from Model, or Model to/from Mediator, and executes simple transforms (i.e., regrid or redistribution)

21

22 of 42

���Community Mediator for Earth Predictive Systems

(CMEPS)���

CMEPS is a NUOPC-compliant mediator for use in CESM and other modeling systems which uses ESMF to couple earth grid components in a hub and spoke system

Coupling between component models�

  • CMEPS is responsible for transferring field information from one model component to another
  • This transfer can require one or more operations on the transferred fields
    • mapping of fields between component grids
    • merging of fields between different components
    • time-averaging of fields over varying coupling periods

23 of 42

CESM Architecture

NUOPC

CMEPS

River Component

​

NUOPC Cap

Land Ice Component

NUOPC Cap

Sea Ice component

NUOPC Cap

Ocean Component

NUOPC Cap

WAVE Component

NUOPC Cap

NUOPC Cap

Atmosphere component

​

Land Component

​

NUOPC CAP

Connectors generated automatically

24 of 42

ESMF/NUOPC Provides Advanced Coupling Capabilities

NUOPC Driver:

  • Data driven run sequence - can easily see lags in model evolution

​

NUOPC Mediator:

  • Parallel online generation of remapping weights - no more mapping files!

​

NUOPC Connectors:

  • Automatic transfer of component data to and from the mediator

​

​

​

Driver

Model:

ATM

Mediator

Model:

SEA ICE

Model:

OCN

Model:

LAND

Model:

LAND ICE

Model:

RIVER

Model:

WAVE

25 of 42

atm/lnd, atm/ice and atm/ocn fluxes

CTSM

Computes atm/lnd fluxes

CICE

Computes atm/ice fluxes

MOM6

Sends SST

Receives atm/ocn fluxes

CMEPS

Maps

Computes

atm/ocn fluxes

Merges

atm/lnd, atm/ice and atm/ocn fluxes before sending to atm

CAM

Receives merged fluxes

  • atm/lnd fluxes computed by land on subgrid tiles and merged to gridcell
  • atm/ice/fluxes computed by ice on subgrid ice categories and merged to gridcell
  • atm/ocn fluxes computed on the atm, ocn or exchange grid in CMEPS (mapping and time averaging of atm/ocn fluxes done in CMEPS)

26 of 42

Benefits of CMEPS – Introducing new Grids

  • (Relatively) easy to introduce new grids
    • No longer need offline mapping files
      • Before: all inter-component mapping files were created offline
        • 25 mapping files needed for a fully coupled pre-industrial control
      • Now: all non-custom mapping files are generated at run time !!!
        • Only 2 mapping files are needed (rof->ocn)
    • Land and ocean fractions on atm/land grid are determined by mapping ocean mask conservatively to land grid
      • Before: each new component grid required generating new offline fraction files and updating CIME configuration files
      • Now: land and ocean fractions are generated at runtime during model initializations!

​

27 of 42

Benefits of ESMF – Introducing new Grids (cont)

  • Easier to introduce new grids – land surface dataset generation is now parallel!
    • Before: needed to create 17 offline mapping files and use these as input to a surface dataset generation code that ran one processor. Took over 2 days to generate a surface dataset at 7.5 km grid.
    • Now: all mapping is done at run time and all I/O is parallel. Now takes 10 minutes to generate a surface dataset for a 7.5 km grid.
    • Now: Enable mapping of 30 second (724M points) soil texture dataset.

​

​

28 of 42

Benefits of CMEPS – New Land-Ice Capabilities

  • Running both Antarctica and Greenland in one simulation (1)
    • Before: proposed approach was to create a unified global grid. New global grids would have to be created for every combination. Results in combinatorial explosion!
    • Now: create a ’nested state’ where each ice sheet in CISM couples to a corresponding ice sheet in the mediator. Very extensible and user friendly approach – new glaciers can be added easily.
    • An arbitrary number of ice sheets can be coupled at run time.

​

  • Enabling Antarctic ocean <-> land-ice coupling (2)
    • Requires regridding ocn->cism fields at multiple levels. Each level has different mask due to different bathymetry.
    • Before: a different mapping file for ocn->cism mapping was required for each ocean level. Each ocean level field was passed separately.
    • Now: can do regridding leveraging ESMF dynamic masking functionality in the mediator. Only one field with multiple levels is passed.

​

29 of 42

Benefits of CMEPS – exchange of 3d fields between components

  • Very easy to exchange 3d fields with ESMF/NUOPC
  • Data in ESMF fields can have an ‘ungridded’ (undistributed) dimension
    • vertical levels
    • multiple 2d fields (e.g. different dust aerosols)
  • Before: needed to specify different field for each different ‘ungridded dimension’
  • Currently this is being used, for example, to send multiple ocean levels to the land-ice component for Antarctic coupling

​

29

30 of 42

Benefits of CMEPS – Greater Computational Efficiency

  • Components sharing cores can now have different threading levels using ESMF-managed threading

​

  • Before:
    • If component A is threaded 4 ways and component B is not threaded, if they are to share the same nodes, component B can only use ¼ of the cores in a node
    • This leads to idle cores and poor HPC resource utilization
  • Now:
    • If component A is threaded 4 ways and component B is not threaded, if they are to share the same nodes, component B can use ALL of the cores in a node
    • This greatly increases the efficiency of the overall model�
  • Pre-industrial, fully coupled run (2º atm/1º ocn)
    • Now Model Cost:             2531  pe-hrs/simulated_year�Now Model Throughput:     35   simulated_years/day�
    • Before Model Cost:             3140   pe-hrs/simulated_year�Before Model Throughput:         31   simulated_years/day�

​

​

​

30

31 of 42

Community Data Models for Earth Prediction Systems (CDEPS)

Data Atmosphere

Ocean

(POP2)

Coupler

(CIME, CDEPS)

Sea Ice

(CICE5)

Biogeochemistry

(MARBL)

​

T, P, Solar Radiation, Longwave Radiation, Humidity, Wind, CO2, etc. from observations (reanalysis)

32 of 42

CDEPS

Introduction

  • CDEPS contains ESMF/NUOPC compliant data components that are modular and flexible: Can be used in any ESMF/NUOPC compliant modeling system
  • CDEPS handles the ability to ingest multiple data sources with different spatial and temporal resolutions. Also provides ability to customize the ingested data (e.g. unit conversions)
  • All data is read with parallel IO (PIO2) – can easily ingest 2d or 3d fields!
  • Automated spatial regridding capability: 1) regridding of 2D/3D fields, 2) support for different regridding types such as conservative, patch, 3) extrapolation.
  • Various time interpolation capabilities (coszen, bilinear, etc)

33 of 42

CDEPS DATM

Introduction

CDEPS within CESM provides many different forcing options out of the box

​

Mediator CMEPS

DATM

BLOM

CICE

CTSM

Driver

Data atmosphere CESM forcings

​

MOM6/CICE/WW3 forcings

  • CORE2
  • JRA
  • ERA5

​

CTSM forcings

  • GSWP3
  • CRU,
  • NLDAS

​

Spinup forcings

  • CPLHIST mode

34 of 42

CDEPS Share Code

  • All data components use same share code for reading in and interpolating forcing data
  • CDEPS inline interface
    • provides an ‘inline’ interface that can be called directly from prognostic components to read in various forcing datasets
    • already used throughout NorESM2.5+/CESM3
  • Advantage of inline interface
    • Can specify only one forcing data at the observational resolution and use that for all component resolutions
    • Do not need to create new forcing datasets for each new component resolutions

35 of 42

CDEPS DOCN

Introduction

Mediator

CMEPS

CAM

DOCN

SST => CAM

Temp3d, Salinity3d => CISM

  • New COMPSET: DOCN%MULTILEV-DOM
  • 2 input forcings – 2d SST and 3d Temp and Salinity
  • Input forcings can be from observational data or mediatory history file
  • NO mapping files needed
  • Destination ice sheets can be set at run time

​

SST

CISM

Greenland and/or Antarctica

Temp3d, Salinity3d

36 of 42

Towards a machine learning enhanced version of CESM (CESM3-MLe)

Learning the Earth with Artificial intelligence and Physics

NSF Science and Technology Center

Harness new ML + data to

transform CESM

​

LEAP forward in the reliability, utility, and reach of climate projections through synergistic innovations in data science and climate science

More Accurate Projections

New ML-based Parameterizations

Next-Generation CESM

Observations

High-Fidelity Models

New ML Algorithms

M2LInES

Schmidt Futures

ML-enabled Parameter Optimization

37 of 42

Towards a machine learning enhanced version of CESM (CESM3-MLe)

Learning the Earth with Artificial intelligence and Physics

NSF Science and Technology Center

M2LInES

Schmidt Futures

Robust and flexible Implementation of ML-based parameterizations requires Fortran-Python bridge

FTorch

ML-parameterization

  • FTorch implementation with CESM working fairly well
  • Needs an integration plan to bring fully into CESM3 infrastructure
  • Documentation
  • Testing, edge case evaluation
  • GPU-CPU combo testing
  • Ideally, some consistency across US modeling centers

38 of 42

Thank you!

​

​

This material is based upon work supported by the National Center for Atmospheric Research, which is a major facility sponsored by the National Science Foundation under Cooperative Agreement No. 1852977.

39 of 42

Workflow

This material is based upon work supported by the National Center for Atmospheric Research, which is a major facility sponsored by the National Science Foundation under Cooperative Agreement No. 1852977.

40 of 42

CMIP7 Forcings

  • Strategy is to test in CESM2-WACCM (underway) - also to provide ozone, N-dep for CMIP7
  • Then, if no issues, test in CESM3-beta
  • CMIP7 forcings for CESM Tracking

41 of 42

CMIP7 Forcings

42 of 42

Quality Control

In-line diagnostics that read history files and output a handful of ‘first-look’ diagnostics (e.g., GHG, solar, RESTOM).

​

Q: CUPiD cntl at specific RESUBMIT time. Store pngs on git or local?

​

Status: WIP (TBD)

Create_newcase

Create_newcase creates a local gitrepo. (CASE_GIT_REPOSITORY: Flag to set remote repo). Documents the run. Provenance, changes, etc.

​

Status: Done

​

Job Control

Opt 1: bash scripting → Nan/Jim/Cecile

Opt 2: ECFlow → Jim Edwards

​

Status: Decision (Jim/Nan/Cecile)

​

Do bash scripting for each step. Then put all these scripts together into ECflow directive for power users.

​

Decisions: Will we handle Large Ensembles differently?

​

CMIP7 Service acct, run on $SCRATCH, not CS

Production

CMIP7 Workflow (Input/forcing data, MIP prep, Production, Post-process, CMORization, Publication)

Workflow slides

Progress tracking