NICAM: Nonhydrostatic Icosahedral Atmospheric Model

Masaki Satoh^1,2

Courtesy of Shuhei Matsugisi¹, Woosub Roh¹, Daisuke Takasuka³

1) Atmosphere Ocean Research Institute, The University of Tokyo

2) Typhoon Research and Technology Center, Yokohama National University

3) Tohoku University

International Summit

Computational System for Frontier Earth System Science and Climate Simulation & Projection

Sunday, September 29 – Wednesday, October 2

I Hotel and Illinois Conference Center, 1900 South First Street, Urbana, IL 61820 USA

Marguerite

Contents

• NICAM overview
• The global highest-resolution (dx=220m) simulation
• The EarthCARE satellite collaboration
• The multi-year climatology experiments
• Proposal of the intercomparison experiments of global storm-resolving models
• NICAM computational performance on Fugaku

​

NICAM: Nonhydrostatic Icosahedal Atmospheric Model

• Development since 2000

Tomita and Satoh(2005, Fluid Dyn. Res.), Satoh et al.(2008, J. Comp. Phys.)

• First global dx=3.5km run in 2004 using the Earth Simulator (JAMSTEC)

Tomita et al.(2005, Geophys. Res. Lett.), Miura et al.(2007, Science)

• High performance computing studies

Further high-resolution (dx=870m) on the K computer (Miyamoto et al. 2014)

Decadal simulations (Kodama et al. 2014; Takasuka et al. 2024)

Ensemble simulations (Yamada et al. 2023)

Global Large Eddy Simulation (dx=220m) on the supercomputer Fugaku (Matsugishi et al. 2024)

• International collaborations
• DYAMOND project: Stevens et al. (2019)
• International Core-to-Core Project on Global Storm Resolving Analysis (2022): Japan, UK, Germany, US
• International Pan-Hackathon, May 2025 (planned)

​

Earth Simulator

京

富岳

The global highest-resolution (dx=220m) simulation

4

• The global highest-resolution (dx=220m) simulation is valuable to knowing the capability of reproducing a realistic atmosphere and the computational performance.
• The global 220-m simulation tends to resolve "large eddies" related to deep convection and is referred to as "Global Large Eddy Simulation (GLES)" for deep convective motions.
• This simulation will be a basis for the more popular "global KM-scale "( global storm-resolving) simulations, and a reference to the intercomparison of the global storm-resolving models (called DYAMOND) for evaluation using the EarthCARE and EC-TOOC observations (June-October 2024).

Satoh, M., Matsugishi, S., 2023: Toward global large eddy simulations. Gekkan Kaiyo, , Vol. 55, No. 4, 172-179, https://doi.org/10.15083/0002007328 (in Japanese)

• The simulation was conducted by using Supercomputer “Fugaku” (R-CCS, RIKEN, Kobe)
• 81,920 nodes, 3,932,160 cores, 2.3 PB Memory (This is > 50% of the total 158,976 nodes).
• Mixed precision version NICAM (Nakano et al. 2017) �327,680 MPI + 12 OMP
• Data size (a variable per a step): 2D: 40GB; 3D:3TB (without compression)

Shuhei Matsugishi

Model settings of NICAM

5

• Horizontal resolution: dx = 3.5km, 1.7km. 870m, 440m, 220m (GL11, 12, 13, 14, 15)
• Vertical level: 78 (dz = 400m in Troposphere)
• Time step: dt = 10, 6, 3, 1.5, 0.75 s for 3.5/1.7/0.87/0.44/0.22 km, respectively
• Integration: Aug. 2016, the DYAMOND-summer condition
• Turbulence scheme: sub km is “Grayzone”: MYNN (Nakanishi & Nino 2004) with Gray zone extension (Ito et al. 2015), Smagorinsky; Cloud microphysics: Single moment 6 category (Tomita 2008; Roh & Satoh 2014); Radiation: MSTRNX(Sekiguchi & Nakajima 2008)

OLR/Precipitation　(Smagorinsky)

6

3.5km (GL11)

0.87km GL13

1.7km (GL12)

220 m (GL15)

440 m (GL14)

Background topography is NASA Blue Marble

Aug. 5, 02Z, 2016

Precipitation (snapshot) Smagorinsky

7

3.5km (GL11)

0.87km GL13

1.7km (GL12)

220 m (GL15)

440 m (GL14)

EarthCARE satellite

Illingworth et al. (2015, BAMS)

CPR

Cloud Profiling Radar

Launch: May, 2024

First images from Cloud Profiling Radar (CPR) onboard Earth Cloud Aerosol and Radiation Explorer (EarthCARE) "Hakuryu"�- World's first measurement of vertical cloud motion from space -

9

Vertical distribution of the radar reflectivity factor by the CPR.

Vertical distribution of Doppler velocity by CPR.

https://global.jaxa.jp/press/2024/06/20240627-1_e.html

June 27, 2024 (JST)

Global precipitation distribution: 06.18.00UTC

GSMAP

GL13

GL11

NICAM 3.5 km

NICAM 870m

EarthCARE

Not shown

GSRM Intercomparison Experiments:

12

Proposed GSRM Intercomparison experiments (GSRMIPs)

• Summer (40 days; DYAMOND-summer; Stevens et al. 2019)
• Winter (40 days; DYAMOND-winter; Miyakawa et al. in prep.)
• 1 season (3 months) (*)
• 1 year for annual cycle (Takasuka et al. 2024)
• 10 years and future time sllice
• 30 years or AMIP & scenario run (NextGEMS)
• Short-term experiment for evaluation with intensive observation (EarthCARE & ORCESTRA; Aug-Sep. 2024)
• Ensemble of extreme case studies (control and pseudo global warming condition)
• 1 year for the annual cycle with + 4K (Terai et al.)
• Patterned Experiment for future TC changes (cf. Zhao and Knutson 2024)

1 year intercomparison experiments�Takasuka et al. (2024, in review)

Computational performance

13

The computational performance of the NICAM simulation on Fugaku.

The resolution dependency of elapse time [s] for one step. From left to right, the grid interval of NICAM is 28, 14, 7.0, 3.5, 1.7, 0.87, 0.44, and 0.22 km.

Expected NICAM computational performance on Fugaku

• For our GLES simulations: L78, single precision, NICAM.21.0,
• gl11(3.5km), 0.105 SYPD, 2560 node, dt=10s
• gl12(1.7km), 0.0196 SYPD, 2560 node, dt=4s
• gl13(870m), 0.00805 SYPD, 10240 node, dt=2s�
• IO is used for about 1/3 of the computation time. If I/O time is omitted, and dt is increased as normal dt=15s, we achieve 0.2SYPD for 3.5km.

​

• For a speed-up run, we use 10240 note, then 0.4SYPD is achieved.�
• Even more, if we can use 40960 nodes, i.e., 25.75% of the total 158,976 nodes of Fugaku, then we expect achieving 1SYPD for 3.5km x 0.1deg NICOCO coupled model.�

Summary

15

• NICAM's recent progress: global 220-m mesh simulation (Shuhei Matsugishi), multi-year simulations (Daisuke Takasuka).
• Proposal of the intercomparison experiments of global storm-resolving models:
• 1-year simulation (present and +4K)
• Short-term simulation to be evaluated by the EarthCARE/ORCESTRA observation (Aug-Sep 2024).
• The 3.5km-mesh NICAM computational performance on Fugaku is 1SYPD if ¼ of Fugaku is used.

Additional slides

Computational performance

17

For the simulation of a horizontal grid spacing of 220 m (GL15) with 78 vertical layers,

• Approximately 10 × 4¹⁵ ≈ 10 billion points on the sphere and 3D variables consist of approximately 800 billion grid points.
• Outputting data in single-precision floating-point format without compression, a 2D snapshot requires 40 GiB (=10 × 415 × 4 bytes), and 3D data requires about 3 TiB.
• If 3D data (temperature, pressure, three wind components, and six hydrometeors: a total of 11 variables) are output every hour for 8 hours (8 snapshots), 264 TiB are needed.

​

• 81,920 nodes are used, which accounts for 51.5% of the total 158,976 nodes of Fugaku.
• 327,680 parallel processes are executed using MPI, combined with 12 OpenMP threads, resulting in a total of 3,932,160 cores used.
• The maximum memory usage is approximately 2 PByte.
• The computational domain is divided into r-level 8, resulting in 655,360 total subdomains, with each MPI parallel process responsible for 2 subdomains.

​

• NICAM mainly uses single-precision and employs mixed precision (Nakano et al. 2018), combining some double-precision calculations to reduce memory usage and increase computational speed.
• The computation speed is about 18 PFlops, with a peak performance ratio of about 7%, demonstrating good weak scaling performance regardless of the resolution.

Top ten reasons why GCRM is a great leap

1. Represents the mesoscale (2 to 2000 km) and its link to the general circulation

2. Represents the multiscale physics and scale interactions inherent in atmospheric moist convection

3. Represents the dynamic and thermodynamic drivers of cloud microphysical processes

4. Represents the mean state to which small scale turbulence responds and captures gravity, and inertial-gravity waves, and thereby main mechanisms of wave mean flow interaction

5. Simulates the same quantities that the satellites observe, enabling the critical application of these observations

6. Provides an additional source of empiricism using “nature runs” with hector or kilometer-scale simulations on short time periods

7. Opens the possibility of global cloud-resolving earth system models by enabling important coupling pathways to the ocean

8. Allows for one model and avoids the need for (and challenges of) downscaling thereby providing a direct link to application and impact

communities

9. Represents main constituent transport mechanisms, thus linking observed concentrations of important trace species to their sources and

sinks

10. Advances information science, by spurring developments in both hardware and software, e.g., to deal with data flows, or hierarchical computational architecture

​

Satoh et al. (2019, Curr. Clim. Change Rep.)

Good points of GSRMs

• Realistic meso-scale circulations, e.g. diurnal cycle, topographic rains
• Multiscale structure of cloud systems
• Intra-seasonal oscillation: MJO, BSISO (boreal summer ISO)
• Tropical cyclones
• Cloud properties with cloud microphysics
• Collaboration with satellite observation
• (evaluation, improvements, & assimilation)

Precipitation zonal mean/histogram

20

mm/day

Diff. from 3.5km

Zonal/Daily mean Precipitation

MYNN

MYNN