ProbSevere – v3

John Cintineo (UW-CIMSS)

Michael Pavolonis (NOAA/NESDIS)

Justin Sieglaff (UW-CIMSS)

SR Science Circle Forecaster Applications Chat

19 April 2022

With support from GOES-R Science

2

What’s the same in ProbSevere v3?

• Multi-platform storm-tracking (satellite and radar)
• Extracts features from MRMS, ENI, GOES-R, NWP
• Uses ML models to create probabilistic guidance for next-hour storm severity
• CONUS-wide
• Updates every 2 min
• Part of MRMS

3

What’s new in version 3?

• PSv3 ML models: gradient-boosted decision trees
• Pros:
• Works well with correlated data
• Better probability calibration
• Usually better performance
• Cons:
• Easier to overfit to training data
• More of a “black box”
• PSv2 ML models: naïve Bayesian classifiers
• Pros:
• Easy to train on small amount of data
• Good generalization
• Cons:
• Poor calibration when predictors not independent

4

Input sources: MRMS, ENTLN, GOES-East, and SPC mesoanalysis

ProbWind (PWv3):

• Max VIL
• Max CompRef
• Max 0-2 km azimuthal shear
• 98^th %-ile 0-2 km azimuthal shear
• Max 3-6 km azimuthal shear
• 98^th %-ile 3-6 km azimuthal shear
• Sat. growth rate (ABI)
• Intense convection prob. (ABI+GLM)
• MeanWind 1-3 km AGL (from RAP)
• Eff. bulk shear (from RAP)
• 0-3 km lapse rate
• Max mid-level lapse rate
• Wet-bulb 0^oC height
• Precipitable water
• Mixed-layer LCL
• DCAPE
• STP (eff. layer)
• Storm-relative wind 0-2 km

ProbHail (PHv3):

• Max MESH
• Max CompRef
• Max Reflectivity at -20^oC
• Max VIL density
• Max 3-6 km azimuthal shear
• 98^th %-ile 3-6 km azimuthal shear
• Max total lightning density
• Sat. growth rate (ABI)
• Intense convection prob. (ABI+GLM)
• Eff. bulk shear (from RAP)
• 0-3 km lapse rate
• Max mid-level lapse rate
• Precipitable water
• Wet-bulb 0^oC height
• CAPE -10^oC to -30^oC
• Storm-relative wind 0-2 km

What’s new in version 3?

5

Input sources: MRMS, ENTLN, GOES-East, and SPC mesoanalysis

What’s new in version 3?

ProbTor (PTv3):

• Max MESH
• Max CompRef
• Max 0-2 km azimuthal shear
• 98^th %-ile 0-2 km azimuthal shear
• Max 3-6 km azimuthal shear
• Intense convection prob. (ABI+GLM)
• Eff. bulk shear (from RAP)
• MeanWind 1-3 km AGL (from RAP)
• 0-3 km lapse rate
• DCAPE
• MLCIN (0-50mb)
• STP (eff. layer)
• Mixed-layer LCL
• Eff. storm-relative helicity
• Storm-relative wind 0-2 km

probSevere (PSv3):

• Max MESH
• Max CompRef
• Max VIL
• Max 0-2 km azimuthal shear
• 98^th %-ile 0-2 km azimuthal shear
• Max 3-6 km azimuthal shear
• 98^th %-ile 3-6 km azimuthal shear
• Max total lightning density
• Max flash-extent density (GLM)
• Sat. growth rate (ABI)
• Intense convection prob. (ABI+GLM)
• MeanWind 1-3 km AGL (from RAP)
• Eff. bulk shear (from RAP)
• 0-3 km lapse rate
• Max mid-level lapse rate
• Wet-bulb 0^oC height
• Precipitable water
• Mixed-layer LCL
• DCAPE
• MLCAPE (0-50mb)
• STP (eff. layer)
• Storm-relative wind 0-2 km

6

IntenseStormNet

• “probability of intense convection”
• Image-based model
• GOES-only
• 0.64-µm reflectance (VIS)
• 10.3-µm BT (window IR)
• GLM flash-extent density
• Its output is a predictor in the PSv3 models.

​

​

Cintineo et al. WAF Dec. 2020

What’s new in version 3?

7

What’s new in version 3?

• Currently using SPC mesoanalysis data
• Moving towards HRRR / Rapid Refresh Forecast System (RRFS)

8

ProbSevere meteogram tool

​

• Double-left-click on PS object (inner contour)

​

• Updates automatically

​

• Defaults to most-recent 2 hours of storm’s history

​

• One window available (i.e., one storm at a time)

What’s new in version 3?

9

How does PSv3 compare to PSv2?

10

How does PSv3 compare to PSv2?

For the “all hazards” models

Trained on 2018-2020; evaluated on 2021

Max CSI

0.26 - 0.28�@ 40-60%

Max CSI

0.24 - 0.26

@ 60-80%

11

How does PSv3 compare to PSv2?

For the “all hazards” models

Trained on 2018-2020; evaluated on 2021

12

Some examples from 2021-2022

13

https://goesrhwt.blogspot.com/2021/06/storm-on-potomac.html

• Low MESH, lightning flash rate
• Good bulk shear

​

• PWv3 = 41% (PWv2 = 3%) 7 min prior to reports of trees down
• VIL and 0-3 km LR top contributors

14

https://goesrhwt.blogspot.com/2021/06/probsevere-v3-gives-sooner-heads-up-in.html

15

- PSv3 = 23% vs. PSv2 = 65% 2 min before SVR warning

- no severe reports

​

- PSv3 is less influenced by ENI lightning

​

Reducing false alarms

16

• PTv3 probabilities much better calibrated
• PTv3 about same max CSI as PTv2
• PTv3 ≥ 60% is very rare
• Max CSI range = 30-40%

​

Reducing false alarms – ProbTor

PTv3

PTv2

For 2021 data

17

https://goesrhwt.blogspot.com/2022/04/salado-tornado.html

PTv3 = 30%

PTv2 = 7%

Top-4 predictors (at 22:14Z):

1. MESH
2. 3-6 km azimuthal shear
3. effective bulk shear
4. IntenseStormNet probability

18

Summary

• PSv3 exhibits higher CSI scores than PSv2, overall
• Largest increases are in low-moderate CAPE and moderate shear regimes

​

• PSv3 is better calibrated and reduced the over-prediction bias in PSv2
• As a result, the most skillful range for PSv3 is 40-60% compared to 60-80% for PSv2, so the warning forecaster should expect to make warning decisions at lower PSv3 values.

​

• PTv3 is better calibrated, about the same max. CSI, compared PTv2
• Working to improve PTv3 with spatial methods (e.g., CNN)
• Most skillful range for PTv3 is 30-40%.

​

• ProbSevere meteogram enables faster diagnosis of storm trends

​

• PSv3 will be evaluated at the 2022 HWT

​

• T2O contingent on forecaster feedback and availability of funding

19

Backup slides

20

PTv3 > PTv2 by 10%+

PTv3 tends to max out much lower (~60% vs. ~90% for PTv2)

PTv3 = 22%; PTv2 = 9%

5 min before warning

Winterset, IA tornado

16 min later, 1st tornado report

21

How does PSv3 compare to PSv2?

For the “all hazards” models

AUPD

For 2021

22

How does PSv3 compare to PSv2?

For the hail models

For 2021

23

How does PSv3 compare to PSv2?

For the wind models

For 2021

24

How does PSv3 compare to PSv2?

For the tornado models

For 2021

25

High-resolution NWP Data

GOES Imagery and Derived Products

MRMS Products

Total Lightning (terrestrial and spaceborne)

Feature extraction

Machine-learning models

Object identification and tracking using GOES ABI and MRMS

Probabilities of severe, hail, wind, tornado