ProbSevere v3 gives sooner "heads up" in South Carolina

Several isolated storms in South Carolina tapped into some better bulk shear, becoming better organized, as well as threatening. ProbSevere v3 (PSv3) highlighted elevated probabilities of severe before version 2 for the three storms shown here. 

Figure 1: Animation of ProbSevere, MRMS MergedRef, and NWS severe weather warnings for several storms in South Carolina yesterday afternoon.

The first storm, northwest of Myrtle Beach, SC (Figures 2 and 3), had PSv3 hovering in the 20-40% range for a while before increasing to 60% and then later to 70%. In the 50 minutes before the official NWS warning, PSv2 was mainly under 10%. The MRMS VIL (32 g/m^2), 0-3 km lapse rate (8.2 C/km) and MRMS 3-6 km AzShear were leading contributors to the PSv3 probability at 19:00 UTC, when PSv3 was about 40% and PSv2 was 8%. 

Figure 2: A storm in eastern SC that downed multiple trees.

Figure 3: PSv3 and PSv2 time series for the storm highlighted in Figure 2.

A second storm, northwest of Charleston, SC, took a while before becoming severe and dropping silver dollar-sized hail. PSv3 remained in the 30-40% range for a while (owing to a favorable environment), while PSv2 was < 10 %. The probabilities in the 30-40% range early on better conveyed the severe threat that this storm would soon exhibit. The VIL (37 g/m^2), 0-3 km lapse rate (8.8 C/km) and the satellite growth rate (moderate) were leading contributors to the enhanced probability of severe early on in this storm's lifetime.

Figure 4: A storm northwest of Charleston, SC, which dropped large hail.

Figure 5: PSv3 and PSv2 time series for the storm highlighted in Figure 4.

A third storm, which followed in the wake of the storm NW of Charleston, SC, also exhibited higher severe probabilities (in the 20-40% range) well before PSv2 latched on to it. This storm went on to produce numerous wind damage reports as well as some large hail. In a similar refrain, the VIL, low-level lapse rate, and satellite growth rate all contributed to the higher probability of severe early on (with the 3-6 km AzShear and composite reflectivity also aiding). The PSv3 models are able to find connections between the observed and environmental predictors in a more robust way, compared to PSv2. 

Figure 6: A third storm in South Carolina, producing numerous wind damage reports.

Figure 7: PSv3 and PSv2 time series for the storm highlighted in Figure 6.

Forecasters at the HWT have been able to use a time series tool in the ProbSevere AWIPSII plug-in. The more accurate ProbSevere v3 models, coupled with instant access to storms' time series history will hopefully aid forecasters in the warning decision-making process. 

ProbHail and ProbWind v3 in Florida

A slow-moving cold front converged with a sea-breeze boundary to produce some strong storms in a weakly-sheared environment. One forecaster at the HWT remarked how the ProbHail values in version 3 (PHv3) seemed more reasonable to him in this environment, compared to ProbHail version 2 (PHv2).

Figure 1: ProbSevere contours, MRMS MergedRef, and NWS severe thunderstorm warnings in the Florida panhandle. The highlighted storm produced severe wind reports

We can see that despite the favorable MESH, satellite growth rate, and thermodynamic parameters in the storm highlighted in Figure 1, PHv3 was only 11% at this time, while ProbWind v3 (PWv3) was 60%. This storm later produced multiple reports of downed trees and power lines. While ProbWind v2 correctly had high probabilities (PWv2 = 81%), PHv2 was heavily overforecasting (PHv2 = 85%). The machine-learning models in ProbSevere v3 (gradient-boosted decision trees) were able to more accurately discriminate between the wind and hail threats in this situation. For this storm, the low effective bulk shear (12 kt), high wetbulb 0C height (13.4 kft) and high PWAT (2.1 in) all detracted from the PHv3 probability. 

Several storms in the Florida panhandle did produce severe hail reports, including the storm highlighted in Figure 2. In general, PHv3 was in the 25-40% range for storms that produced hail in this environment, and perhaps could help forecasters identify hail threats compared to other storms in the area not producing hail, which had PHv3 probabilities in the 0-15% range. The higher MESH (1.5 in) and slightly better effective shear (19 kt) contributed to higher PHv3 for this storm.

Figure 2: ProbSevere contours, MRMS MergedRef, and NWS severe thunderstorm warnings in the Florida panhandle. The highlighted storm produced severe hail and wind reports.

Northern Minnesota storms

At the 2021 HWT, one team of forecasters was working in eastern North Dakota and northern Minnesota yesterday. After monitoring a string of very weak-looking storms, a few storms finally tapped into some better deep-layer shear and overcame a capped environment.

One forecaster noted that a storm in Cass County looked severe, and while it was just outside of their county warning area, they would have warned it for 1" hail and 60-mph winds. The storm showed a small hail spike at 21:30 UTC (Figure 1). 

Figure 1: A small storm in Cass County, MN, with a hail spike. 

The team of forecasters also noted how PSv3 seemed to handle the marginally severe nature of this storm and others in the area better than PSv2, with PSv3 exhibiting higher probabilities earlier and maintaining them better than PSv2. 

Figure 2: ProbSevere contour with hover-readout and time series window, and MRMS MergedRef for the storm in Cass County. 

At 21:46, PWv3 was 49% whereas PWv2 was 11%. The MRMS VIL (30.3 kg/m^2), 0-3 km lapse rate (8.1 C/km), GOES intense convection probability (ICP; 24%), and 1-3 km mean wind were the top four contributors to the higher probability of wind for this storm

Further northwest, another storm showed a wind threat. PSv2 and PSv3 were fairly similar for this warned storm, which later produced a 61-mph wind gust near Red Lake. 15 minutes before the wind report, PSv3 achieved 67%, with the VIL, 0-3 km lapse rate, ICP, and 1-3 km mean wind contributing the most to the probability. 

Figure 3: ProbSevere contours, MRMS MergedRef, and official NWS severe weather warnings. 

Forecasters have remarked how they would use the time series feature in the ProbSevere AWIPS plug-in, if available at their offices. They also desire new enhancements, such as meteograms of more predictors and the ability to "dock" the window within a CAVE pane or tab. 

Storm on the Potomac

Moderate instability and good deep-layer shear produced a line of storms near a surface trough in northern Virginia. ProbWind v3 was handling the severe threat better than v2, with a probability of 41% vs. 3%, seven minutes before reports of trees down. PWv3 increased by over 20% at 17:00 UTC, due to increases in the MRMS VIL and 0-3 km lapse rate, with the MRMS azimuthal shears and composite reflectivity also contributing. Forecasters at the HWT have noted this week that PWv3 seems to be better calibrated to the wind threat than its predecessor.

ProbWind in northern Alabama

At the HWT, forecasters working the warning desk in Jackson, MS noted an arc of storms in northern Alabama where ProbWind v3 was much higher than its v2 counterpart. They shared that the greater ProbWind probabilities and the fact that they received sub-severe LSRs, (~40 kt gusts) gave them more confidence that there could be severe-level reports soon. The NWS in Huntsville, AL issued a severe thunderstorm warning at 21:18 UTC and there were indeed trees down at 21:49 UTC, near Athens, AL.

Figure 1: ProbSevere v3 contours, MRMS MergedRef, and official NWS severe weather warnings for storms in northern AL.

Figure 2: ProbSevere time series for the storm highlighted in Figure 1.

ProbWind v3 produced greater and more consistent probabilities than v2 for this storm. A post-mortem analysis showed that the top-5 contributing predictors were:

1. MRMS VIL (23.6 J/kg)
2. 0-3 km lapse rate (7.6 C/km)
3. MRMS 0-2 km AzShear (7 x 0.001 /s)
4. MRMS 3-6 km AzShear (7 x 0.001 /s)
   
5. ABI+GLM intense convection probability (ICP; 69%).

The ProbSevere team has been in active discussions with HWT forecasters regarding explaining and conveying model predictions in AWIPS in near-realtime. 

Big storms; small storms

While there were a number of storms during the Central Plains severe weather outbreak on May 26th, one long-lived supercell takes the cake. It persisted for more than 8 hours, dropping giant hail (up to 4" in diameter) and several tornadoes from Hays to Salina, Kansas.

ProbTor version 3 (PTv3) gave much more consistent guidance than version 2, with fewer large fluctuations before tornadogenesis. At 19:32 UTC, about 25 minutes before the first tornado report, PTv3 was at 38% while PTv2 was 11%. At this time, the MRMS azimuthal shears and MESH, SPC significant tornado parameter (> 2), and the GOES intense convection probability (ICP) were leading contributors to the higher ProbTor probability. The ICP is a deep-learning model using GOES ABI + GLM input images. The ICP is a predictor in each PSv3 model. You can see the ICP plotted around this storm in Figure 3, along with ABI imagery and local storm reports. You can also interrogate time series for this storm, saved here. 

Figure 1: ProbSevere contours (outer contour is for ProbTor value), MRMS MergedRef, and NWS severe weather warnings for a supercell in central Kansas.  

Figure 2: Time series of PSv3 models for the storm in Figure 1. 

Figure 3: ICP contours and local storm reports for a storm in central Kansas.

Even though the big storms on the Plains usually get all of the attention, severe weather was ongoing elsewhere. In Akron, Ohio, for instance, a storm in a more marginal environment (30 kt eff. shear; 700 J/kg MUCAPE) downed numerous trees. At 17:06 UTC, about 30 minutes before the first reports of downed trees, PSv3 was at 62% while PSv2 was 24%. The strong mean wind 1-3 km AGL (33 kts), moderate ENI lightning density (0.66 fl/km^2/min), favorable 0-3 km lapse rate (8.2 C/km) and MRMS azimuthal shears were the highest contributors to PSv3 at this time. 

Figure 4: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings for a storm near Akron, Ohio.

Figure 5: Time series of PSv3 and PSv2 for the storm in Figure 4. 

And yesterday, a storm in far southern Illinois damaged mobile homes near Vienna. PSv3 was picking up on this storm much better than PSv2, with a probability of 46% about 15 minutes before the report (PSv2 was 3%). The MRMS VIL (27 kg/m^2), low-level lapse rate (7.9 C/km), ENI lightning density (0.34 fl/km^2/min), MRMS 3-6 km azimuthal shear (moderate), and satellite growth rate (moderate) were the top contributors. 

We hope these examples illustrate some of the improvements users can expect to see with PSv3.

Figure 6: ProbSevere contours and MRMS MergedRef for a storm in southern Illinois.

Figure 7: Time series of PSv3 and PSv2 for the storm in Figure 6.

A note on ProbSevere calibration

ProbSevere v3 (PSv3) models are gradient-boosted decision tree classifiers, which generally produce better calibration of probabilities (i.e., the probability values better match the frequency of reports) than the naive Bayesian classifiers of ProbSevere v2 (PSv2). So, forecasters will aptly observe lower probabilities in PSv3, in general. 

The models are trained, validated, and calibrated against NCEI Storm Data reports. Reports from this database are matched up with ProbSevere objects, representing the "truth" or "labels" of the dataset. It is well-known that Storm Data has reporting biases and artifacts, but is still generally regarded as the best nationwide severe-weather-reporting database. What this all means is that while PSv3 models are very well calibrated to Storm Data reports, they may underforecast actual severe weather occurrence in some cases (this is because Storm Data reports are only a subset of all actual severe weather). 

We have seen underforecasting occur in some hail-producing storms. Here is an example in northern Texas. A supercell produced numerous hail reports (up to 3" in diameter). PSv3 topped out at about 80%, whereas PSv2 was > 95%. This was a no-doubt-about-it hailer, with MRMS MESH exceeding 3" briefly (Figures 1 and 2). 

Figure 1: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings (red and yellow polygons) for a storm near Spearman, Texas.

Figure 2: Time series os PSv3 and PSv2 for the storm in Figure 1. 

Here is another example on the Kansas / Colorado line. This storm was warned for several hours, achieved a maximum MRMS MESH of 1.8", yet never resulted in any reports (based on the SPC log). Though the population density is low in this region, severe hail was reported on storms just to the north and east of this storm. PSv3 was generally 20-30% less than PSv2 throughout the storm's history. It's certainly possible that a storm like this actually produced severe hail, but it simply went unreported. If that was the case, storms like this could dilute the severe class during training of the models, affecting model calibration. 

Figure 3: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings (red and yellow polygons) for a storm near Coolidge, Kansas.

Figure 4: Time series os PSv3 and PSv2 for the storm in Figure 3.

So, practically speaking, forecasters should expect lower probabilities for PSv3 compared to PSv2, and mental "warning thresholds" may need to be adjusted (e.g., "30% is the new 50%"; "60% is the new 80%"). The improved calibration also resulted in more skillful models, not just lower probabilities; there are many fewer false alarms and a number of examples where PSv3 is correctly 20-40% greater than PSv2. We hope this helps users to understand ProbSevere's calibration better and ultimately aid in its utility during warning operations. 

Long-lived storm in Illinois

A long-lived, periodically tornadic storm churned through Illinois yesterday. The storm formed ahead of an approaching cold front and produced large hail and severe straight-line wind gusts along with tornadoes. Tornadoes touched down west and east of Springfield, Illinois.

ProbTor v3 (PTv3) demonstrated a better handle on the tornadic threat than ProbTor v2 (PTv2). PTv3 uses gradient-boosted decision trees to predict the probability of a tornado in the near-term for a given storm. It incorporates more NWP, MRMS, and GOES-16 fields than PTv2. 

A predictor analysis at 21:40 UTC showed that the MRMS 0-2 km and 3-6 km azimuthal shears were highly contributing to the probability, which was 34% at this time (PTv2 was 6%). The significant tornado parameter [effective layer] (STP) was also helping, with a value of 0.8. The effective storm-relative helicity (ESRH) was about 140 J/kg, whereas the 0-1 km SRH was only 85 J/kg, which could be one reason PTv2 was rather low, as PTv2 does not use the ESRH as a predictor. 

At the HWT in 2021, forecasters will also be able to see any given storm's ProbSevere history with the a double-click of the storm object in AWIPSII. We hope this will improve users' situational awareness and help with warning-decision making. 

Figure 1: ProbSevere (colored contours) and MRMS MergedRef for a storm in central Illinois. Inset shows the history of the storm's ProbSevere probabilities. 

Typically, PTv3 is lower than PTv2 values, as a result of improved probability calibration in PTv3 (e.g., PTv2 was sometimes "too hot" and over-predicted the threat). But in this case, PTv3 better highlighted the tornadic threat with greater probabilities than v2.

Figure 2: Time series of ProbSevere v3 models. 

Windy in Philly

A modest-looking thunderstorm downed numerous trees and power lines in the Philadelphia area yesterday evening. This storm resided in a high-shear low-CAPE environment (50 kt and 500 J/kg, respectively). 

Figure 1: ProbSevere (storm contours), MRMS MergedRef, and NWS severe weather warnings (yellow polygons) for a storm in the Philadelphia area. 

ProbSevere v3 (PSv3), was able to get a better handle on this storm than v2. PSv3 increased to 28% at 23:40 UTC, the time of the first report. This increase was due to a favorable environment (eff. bulk shear = 53 kt; Meanwind 1-3 km AGL = 41 kt; STP = 0.6, 0-3 km lapse rate = 6.8 C/km) and increasing lightning density (though the storm still had a low flash rate, overall). 

At 23:54 UTC, PSv3 was 41%, with MRMS azimuthal shears increasing modestly. Though the probabilities were somewhat low, PSv3 showed a good improvement over PSv2, which gave probabilities of < 10% almost throughout. 

Forecasters will be able to evaluate PSv3 this spring at the HWT. An offline analysis of PSv3 has found it better calibrated and more skillful that v2, overall. 

Figure 2: Time series of PSv3, PSv2, NWS warnings, and severe local storm reports. 

ProbSevere time series tool

At the 2021 HWT, forecasters will be able to use a new feature of the ProbSevere AWIPS plug-in: a time series tool. Forecasters from previous HWTs have consistently given positive feedback on a web-based meteogram tool, and so we have implemented something similar in AWIPS. 

To use it, you simply double-click on a ProbSevere object, and a window opens up with the time series of ProbHail, ProbWind, ProbTor, and ProbSevere (prob. of any hazard) for the given storm. We hope this will help forecasters better monitor the trends in hazard probabilities.

Figure 1: A severe-hail-producing storm in northeastern North Carolina, and the associated history of its ProbSevere probabilities in a time series window (ProbSevere was equal to ProbHail for this storm).

ProbSevere v3 will also be demonstrated at this year's HWT. PSv3 is driven by a new statistical model (gradient-boosted decision trees) and incorporates new MRMS, ABI, GLM, and SPC mesoanalysis data. This storm was warned at 17:31 UTC, and produced 1-inch hail at 17:38 UTC. At 17:26 UTC, ProbHail v3 jumped to 25%, whereas v2 was only 4%. ProbHail v3 might have been able to highlight this strengthening storm to the forecaster, whereas version 2 did not. A predictor importance analysis of ProbHail v3 for this storm at 17:26 UTC revealed that the highest contributing predictors were:

1. MRMS reflectivity at -20C (52 dBZ)

2. Eff. bulk shear (40 kt)

3. MRMS composite reflectivity (66 dBZ)

4. MRMS MESH (0.55 in)

5. Wet-bulb 0C height (7180 ft)

We expect that ProbSevere v3 will be more accurate and better calibrated than ProbSevere v2, meaning the probabilities more closely match severe report occurrence.