1 of 50

What’s Past is Prologue:

Studies of Extreme Weather using Machine Learning �and Climate Emulators

William D. Collins and Ankur Mahesh

Berkeley Lab and UC Berkeley

NERSC & NVIDIA

HENS Part 1

HENS Part 2

HENS Part 1

HENS Part 2

2 of 50

Outline

Low-likelihood High Impact Extremes (LLHIs)

Background: data-driven forecasting�
FourCastNet model

Forecast results on extremes
Architecture & training strategy�

Ongoing/future work

Construction of Huge Ensembles (HENS)
Quantifying value added by HENS

3 of 50

Low-Likelihood High Impact Extremes

3

IPCC AR6:

… In summary, the future occurrence of LLHI events linked to climate extremes is generally associated with low confidence, but cannot be excluded, especially at global warming levels above 4°C.

Types of LLHIs to be investigated

Heatwaves
Atmosphere rivers
Tropical cyclones and hurricanes

4 of 50

Low-Likelihood High Impact Extremes

4

5 of 50

Why LLHIs? From the IPCC AR6 WG1 SPM….

6 of 50

Deterministic Numerical Weather Prediction (NWP)

Solving equations of motion for an incompressible fluid on a rotating sphere

HPC-intensive: European Center for Medium Range Weather Forecasting

7 of 50

X 10 Billion

V. Balaji / GFDL

Faster computers have bridged 4 orders of magnitude, out of 16

8 of 50

The slow but steady evolution of classical NWP

Data assimilation and other advances have slowly revolutionized the quality & accuracy over decades

Bauer et al., Nature, 2015

Caveat: These advances are the foundation for assimilated state estimates that ML approaches rely on.

9 of 50

Gains in Resolution for Climate Modeling

9

10 of 50

Climate Phenomena and Models are Inherently Multiscale

11 of 50

Fully Data-Driven Weather Prediction (DDWP)

Mining the data assimilated states from operational NWP
# training samples = length of satellite record (~ 15k days)
Can be stood up by small teams within tech companies.
Is producing skill gains rapidly.

A modern alternative, intriguingly complementary

12 of 50

Deuben & Bauer (2018), 6° , 60x30, 1.8K pixels, MLP

WeatherBench, Rasp et al. (2020). 5.625°, 64x32, 2K pixels, CNN

Weyn et al. (2019), 2.5° N.H only, 72x36, 2.6k pixels, ConvLSTM

DLWP, Weyn et al. (2020). 2°, 16K pixels, Deep CNN on Cubesphere/(2021) ResNet

FourCastNet, Pathak et al. (2022), 0.25°, ~1,000,000 Pixels, ViT+AFNO

GNN, Keisler et al. (2022), 1°, 64,000 Pixels, Graph Neural Networks

FourCastNet: DDWP at very high resolution

13 of 50

FourCastNet pushing the frontier of AI-Driven Digital Twins

Scope Global
Model Type Full-Atmosphere AI Surrogate
Architecture Fourier Neural Operator
Resolution 25km, 6-hourly
Training Data ERA5 Reanalysis
Initial Condition ERA5 / GFS / UFS
Training Time 1000 GPU-hrs
Inference Time 3 sec (2-week forecast)
Calibration IC + Bayesian model uncertainty
Speedup vs NWP O(10,000 – 100,000)
Power Savings O(10,000)
Max Stable Rollout Years
Project Type Open-source

14 of 50

FourCastNet (Fourier foreCasting Network)

Pathak et al. (2022), FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators, arXiv:2202.11214

Extending to include radiation processes, vapor transport, clouds

Training set: 1979 to 2015

Validation set: 2016, 2017

Held out: 2018 onwards

Purely data-driven ML surrogate weather model
Fourier transform for global convolution
Learns solution operator, mesh and resolution invariant
Training data: ERA5 reanalysis
Autoregressive time interval: 6 hours
73 state variables selected:

Temperatures, winds, geopotential & humidity (surface & 12 vertical levels)
Surface pressure, column water vapor, …

15 of 50

FourCastNet and other DDWP models available online

Modulus-Makani, Earth2-MIP, ECMWF AI Lab

https://github.com/NVIDIA/earth2mip

https://github.com/NVIDIA/modulus-makani

Open sourcing and rapid training are indispensable for academic partnerships
Our collaboration have actively contributed to these code bases.

16 of 50

2021: 100 petaFLOPS/s (NERSC)

2022: ~1.7 exaFLOPS/s (OLCF)

Enabling Technology: DOE’s GPU Exascale Systems

2023: ~1 exaFLOPS/s (ALCF)

17 of 50

For simulating LLHIs, DDWP is a “killer app”

Characterizing LLHIs requires examination of many realizations of them.

The only way to produce a sufficient number of realizations is via simulation.

Given the low frequency of LLHIs, the only way to conduct these simulations is with an emulation of NWP/ESM codes that runs orders of magnitude faster.

This is why DDWP is valuable for characterizing rare climate extremes.

18 of 50

”Must-have” use case for DDWP for Extremes

Proposal: Generate statistics on simulated LLHIs that could have occurred under historical conditions, as well as their drivers, by generating ��

HENS: Huge Ensembles of 10^N members, �where N > 4 required to converge statistics

The ensemble will consist of short (2-week long) hindcasts.

Hypothesis: Ergodicity of climate system means we can ”trade” �increasing ensemble size with increasing length of sampling time.

19 of 50

HENS in context of increasing ensemble size

20 of 50

Proof-of-Principle Huge Ensembles from FourCastNet

99th%ile

Observed 2018 temp

NVIDIA: https://www.youtube.com/watch?v=FUUT6IrQjo4

Some of Africa’s hottest reliably measured temperatures were encountered in Algeria in �July 2018.

In this proof-of-principle demonstration of huge ensembles, David Hall (NVIDIA) showed that these can emulate extreme temperatures.

21 of 50

It’s the AI-powered sequel to “Groundhog Day”

22 of 50

Constructing HENS: We construct ensembles with FourCastNet using the same ensembling techniques as operational weather centers

Validating HENS: We validate these ensembles on extremes using the same techniques as NWP

LLHIs in HENS: Summer 2023 was the hottest summer on record. We will study and quantify near-miss LLHIs in ultra-large counterfactuals of summer 2023.

Huge Ensembles (HENS) for Summer 2023

Source: NASA Earth Observatory

During this talk, we want to highlight three aspects of our progress creating huge ensembles (HENS) with FourCastNet.

First, we will provide an overview of the methods used to construct a huge ensemble of FourCastNet. These methods are grounded in decades of research in operational meteorology and numerical weather prediction.

Second, we will validate ensembles of FourCastNet using the same tests and diagnostics that operational weather centers use.

Third, we study low-likelihood high-impact events in the summer of 2023. This past summer was the hottest summer on record, surpassing the previous record by 0.23 degrees Celsius. With this increase in the average temperature, we will use HENS to explore "How many more LLHIs could there have been?” In the ultra-large ensemble of counterfactual weather, we will study near-miss extremes that occur within the two-week predictability limit.

23 of 50

Validation of HENS using Metrics from NWP

We validate FourCastNet using the same metrics used by operational weather centers.

Can the forecast distinguish between extremes and non-extremes?

Do forecast probabilities of extremes match their observed occurrence?

Does the ensemble spread match its skill?

Does the distance between ensemble and climatology match numerical models?

Are the power spectra realistic?

24 of 50

ROC curves are plots of the hit rate versus the false alarm rate.

The area under the curves is a scalar metric of the fidelity of the forecasts.

An area of 1 implies perfect forecasts.

An area of 0.5 reflects random forecasts.

Validation #1: Receiver Operating Characteristic

Receiver Operating Characteristic Curve for Total Column Water Vapor at 2 day lead time. Results are shown for the average of June 2023. An extreme event is defined as any event surpassing the the climatological 95th percentile.

Successful forecasts form a right angle with 0 false positive rate and 1.0 true positive rate.

Random Forecast

25 of 50

Validation #1: ROC Integral vs. Time

Random Forecast

26 of 50

Validation #2: Reliability Diagrams

27 of 50

Validation #2': Continuous Ranked Probability Score

28 of 50

Growth of Ensemble Mean RMSE matches NWP

An ensemble of FourCastNet models nearly matches IFS’s performance on ensemble mean RMSE.

29 of 50

Validation 3: Spread/RMSE Ratio Should Approach 1

30 of 50

Validation 4: Measure Gap between HENS & Climatology

31 of 50

Validation 4: Extreme Forecast Index

EFI measures the distance between the ensemble forecast and the model climatology

32 of 50

Correlation between HENS EFI and IFS EFI

Whole summer 2023: 2m Surface Air Temperature

33 of 50

Validation 5: Realistic (Unsmoothed) Power Spectra

34 of 50

Workflow and Early Results from HENS

DaVinci AI

35 of 50

Constructing Huge Ensembles with FourCastNet

To construct the ensemble, we use the same ensemble generation techniques as NWP models.

Ensemble weather forecasts leverage the butterfly effect. The atmospheric is a chaotic dynamical system that displays sensitive dependence on initial conditions. To generate an ensemble of possible future atmospheric states, we perturb the initial conditions. Our initial condition perturbation method is inspired by the IC perturbation methods used by centers such as ECMWF and NCEP have in this field.

In addition to perturbing the input fields, we also perturb the model itself. This is also analogous to the NWP’s use of perturbed parameter ensembles.

Finally, we inject noise during the model integration. The NWP uses a similar method, in which they perturb the tendencies output by the parameterized scheme in the weather model.

However, the European Center for Medium-range Weather Forecasts generates 50-members ensembles. We use these same ensembling techniques to generate ensembles that are two orders of magnitude bigger.

36 of 50

Number of checkpoints is set by asymptotic spread

To construct the ensemble, we use the same ensemble generation techniques as NWP models.

Ensemble weather forecasts leverage the butterfly effect. The atmospheric is a chaotic dynamical system that displays sensitive dependence on initial conditions. To generate an ensemble of possible future atmospheric states, we perturb the initial conditions. Our initial condition perturbation method is inspired by the IC perturbation methods used by centers such as ECMWF and NCEP have in this field.

In addition to perturbing the input fields, we also perturb the model itself. This is also analogous to the NWP’s use of perturbed parameter ensembles.

Finally, we inject noise during the model integration. The NWP uses a similar method, in which they perturb the tendencies output by the parameterized scheme in the weather model.

However, the European Center for Medium-range Weather Forecasts generates 50-members ensembles. We use these same ensembling techniques to generate ensembles that are two orders of magnitude bigger.

37 of 50

Parameters and Contents of HENS

The parameters of HENS are:

Number of models	Number of initial conditions
29	256

HENS consists of 29*256=7424 15-day hindcasts started from each summer day:

6/1/23

6/2/23

6/15/23

6/16/23

9/14/23

8/31/23

…

7424 members

…

38 of 50

Computation of HENS

Parameter	Value
# of nodes / GPUs	64 / 256
Run time for 1 start day	~45 minutes
Run time for 1 season
GPU-hours	69 * 256 = 17664 hrs = 2 yrs

Perlmutter Supercomputer

Perlmutter All-Flash Scratch

Project Disk

8 GB/sec

10 GB/sec

Model Field	Meaning
t2m	2-meter surface air temperature
t2d	2-meter surface dewpoint temperature
tcwv	Total column water vapor
t850	Air temperature at 850 hPa
q850	Specific humidity at 850 hPa
z500	Geopotential height at 850 hPa
msl	Mean sea level pressure

24 TB/start day

2.2 PB/season

39 of 50

29 Model Variants are Statistically Indistinguishable

40 of 50

Information on Extremes Gained from Huge Ensembles

41 of 50

Extreme Information Gain Applies at the Grid-point Level

42 of 50

HENS produces more confident hindcasts

An extreme forecast is issued by binarizing each ensemble member

as "extreme" or "not extreme," using the 99th percentile 2m temperature at each location.

The extreme temperature forecast is the percent of ensemble members that are above the threshold.

43 of 50

Does HENS capture events outside range of IFS ensemble?

For the ensembles initialized throughout summer, at a 10 day lead time (240-258 hours), the HENS ensemble range includes events missed by the IFS ensemble range.

For these IFS misses, the maximum HENS member is greater than ERA5. Most of the distribution is above the 1:1 line. (right)

44 of 50

Next: Do Tropical Cyclones/Hurricanes Obey Classical Diffusion?

HENS creates 1000s of copies of single storms

Let’s Compute How Dora’s Forecast Tracks Spread

The tracks spread like

<x²>=t^Ewith E=2.

Classical E=½

45 of 50

Private-Sector Applications of HENS: Group Insurance Group AXA

46 of 50

Looking ahead: Filling the gap between weather (short-term) and climate (long-term).

Weather Climate

47 of 50

Next Steps and Future Directions for ML Emulators

Extending emulators to all components of the Earth system

Developing extrapolation methods to “no analog” climates

Introducing data-driven climate prediction to the IPCC�(Intergovernmental Panel on Climate Change)

Integrating data-driven weather prediction with �existing operational forecasting

Demonstrating the real-world benefits from emulation

48 of 50

Thanks to all collaborators!

Mike Pritchard

Ankur Mahesh

Boris

Bonev

Josh

North

Noah

Brenowitz

Travis

O’Brien

Yair

Cohen

David

Pruitt

Peter

Harrington

Mark

Risser

Karthik

Kashinath

Shashank

Subramanian

Thorsten

Kurth

Jared

Willard

Amanda

Bowden

Paul

Goddard

Kwesi

Quagraine

Abdoul

Zeba

49 of 50

2025: 1^st GRC on ML for Actionable Climate Science

50 of 50

Acknowledgements:�

This research was supported by the Director, Office of Science,

Office of Biological and Environmental Research of the

U.S. Department of Energy under Contract No. DE-AC02-05CH11231

and by the Regional and Global Model Analysis Program area within

the Earth and Environmental Systems Modeling Program.

The research used resources of the National

Energy Research Scientific Computing Center

(NERSC), also supported by the Office of Science

of the U.S. Department of Energy,

under Contract No. DE-AC02-05CH11231.

Questions?

HENS Part 1

HENS Part 2