Big storms; small storms

While there were a number of storms during the Central Plains severe weather outbreak on May 26th, one long-lived supercell takes the cake. It persisted for more than 8 hours, dropping giant hail (up to 4" in diameter) and several tornadoes from Hays to Salina, Kansas.

ProbTor version 3 (PTv3) gave much more consistent guidance than version 2, with fewer large fluctuations before tornadogenesis. At 19:32 UTC, about 25 minutes before the first tornado report, PTv3 was at 38% while PTv2 was 11%. At this time, the MRMS azimuthal shears and MESH, SPC significant tornado parameter (> 2), and the GOES intense convection probability (ICP) were leading contributors to the higher ProbTor probability. The ICP is a deep-learning model using GOES ABI + GLM input images. The ICP is a predictor in each PSv3 model. You can see the ICP plotted around this storm in Figure 3, along with ABI imagery and local storm reports. You can also interrogate time series for this storm, saved here. 

Figure 1: ProbSevere contours (outer contour is for ProbTor value), MRMS MergedRef, and NWS severe weather warnings for a supercell in central Kansas.  

Figure 2: Time series of PSv3 models for the storm in Figure 1. 

Figure 3: ICP contours and local storm reports for a storm in central Kansas.

Even though the big storms on the Plains usually get all of the attention, severe weather was ongoing elsewhere. In Akron, Ohio, for instance, a storm in a more marginal environment (30 kt eff. shear; 700 J/kg MUCAPE) downed numerous trees. At 17:06 UTC, about 30 minutes before the first reports of downed trees, PSv3 was at 62% while PSv2 was 24%. The strong mean wind 1-3 km AGL (33 kts), moderate ENI lightning density (0.66 fl/km^2/min), favorable 0-3 km lapse rate (8.2 C/km) and MRMS azimuthal shears were the highest contributors to PSv3 at this time. 

Figure 4: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings for a storm near Akron, Ohio.

Figure 5: Time series of PSv3 and PSv2 for the storm in Figure 4. 

And yesterday, a storm in far southern Illinois damaged mobile homes near Vienna. PSv3 was picking up on this storm much better than PSv2, with a probability of 46% about 15 minutes before the report (PSv2 was 3%). The MRMS VIL (27 kg/m^2), low-level lapse rate (7.9 C/km), ENI lightning density (0.34 fl/km^2/min), MRMS 3-6 km azimuthal shear (moderate), and satellite growth rate (moderate) were the top contributors. 

We hope these examples illustrate some of the improvements users can expect to see with PSv3.

Figure 6: ProbSevere contours and MRMS MergedRef for a storm in southern Illinois.

Figure 7: Time series of PSv3 and PSv2 for the storm in Figure 6.

A note on ProbSevere calibration

ProbSevere v3 (PSv3) models are gradient-boosted decision tree classifiers, which generally produce better calibration of probabilities (i.e., the probability values better match the frequency of reports) than the naive Bayesian classifiers of ProbSevere v2 (PSv2). So, forecasters will aptly observe lower probabilities in PSv3, in general. 

The models are trained, validated, and calibrated against NCEI Storm Data reports. Reports from this database are matched up with ProbSevere objects, representing the "truth" or "labels" of the dataset. It is well-known that Storm Data has reporting biases and artifacts, but is still generally regarded as the best nationwide severe-weather-reporting database. What this all means is that while PSv3 models are very well calibrated to Storm Data reports, they may underforecast actual severe weather occurrence in some cases (this is because Storm Data reports are only a subset of all actual severe weather). 

We have seen underforecasting occur in some hail-producing storms. Here is an example in northern Texas. A supercell produced numerous hail reports (up to 3" in diameter). PSv3 topped out at about 80%, whereas PSv2 was > 95%. This was a no-doubt-about-it hailer, with MRMS MESH exceeding 3" briefly (Figures 1 and 2). 

Figure 1: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings (red and yellow polygons) for a storm near Spearman, Texas.

Figure 2: Time series os PSv3 and PSv2 for the storm in Figure 1. 

Here is another example on the Kansas / Colorado line. This storm was warned for several hours, achieved a maximum MRMS MESH of 1.8", yet never resulted in any reports (based on the SPC log). Though the population density is low in this region, severe hail was reported on storms just to the north and east of this storm. PSv3 was generally 20-30% less than PSv2 throughout the storm's history. It's certainly possible that a storm like this actually produced severe hail, but it simply went unreported. If that was the case, storms like this could dilute the severe class during training of the models, affecting model calibration. 

Figure 3: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings (red and yellow polygons) for a storm near Coolidge, Kansas.

Figure 4: Time series os PSv3 and PSv2 for the storm in Figure 3.

So, practically speaking, forecasters should expect lower probabilities for PSv3 compared to PSv2, and mental "warning thresholds" may need to be adjusted (e.g., "30% is the new 50%"; "60% is the new 80%"). The improved calibration also resulted in more skillful models, not just lower probabilities; there are many fewer false alarms and a number of examples where PSv3 is correctly 20-40% greater than PSv2. We hope this helps users to understand ProbSevere's calibration better and ultimately aid in its utility during warning operations.