ProbWind in northern Alabama

At the HWT, forecasters working the warning desk in Jackson, MS noted an arc of storms in northern Alabama where ProbWind v3 was much higher than its v2 counterpart. They shared that the greater ProbWind probabilities and the fact that they received sub-severe LSRs, (~40 kt gusts) gave them more confidence that there could be severe-level reports soon. The NWS in Huntsville, AL issued a severe thunderstorm warning at 21:18 UTC and there were indeed trees down at 21:49 UTC, near Athens, AL.

Figure 1: ProbSevere v3 contours, MRMS MergedRef, and official NWS severe weather warnings for storms in northern AL.

Figure 2: ProbSevere time series for the storm highlighted in Figure 1.

ProbWind v3 produced greater and more consistent probabilities than v2 for this storm. A post-mortem analysis showed that the top-5 contributing predictors were:

1. MRMS VIL (23.6 J/kg)
2. 0-3 km lapse rate (7.6 C/km)
3. MRMS 0-2 km AzShear (7 x 0.001 /s)
4. MRMS 3-6 km AzShear (7 x 0.001 /s)
   
5. ABI+GLM intense convection probability (ICP; 69%).

The ProbSevere team has been in active discussions with HWT forecasters regarding explaining and conveying model predictions in AWIPS in near-realtime. 

Big storms; small storms

While there were a number of storms during the Central Plains severe weather outbreak on May 26th, one long-lived supercell takes the cake. It persisted for more than 8 hours, dropping giant hail (up to 4" in diameter) and several tornadoes from Hays to Salina, Kansas.

ProbTor version 3 (PTv3) gave much more consistent guidance than version 2, with fewer large fluctuations before tornadogenesis. At 19:32 UTC, about 25 minutes before the first tornado report, PTv3 was at 38% while PTv2 was 11%. At this time, the MRMS azimuthal shears and MESH, SPC significant tornado parameter (> 2), and the GOES intense convection probability (ICP) were leading contributors to the higher ProbTor probability. The ICP is a deep-learning model using GOES ABI + GLM input images. The ICP is a predictor in each PSv3 model. You can see the ICP plotted around this storm in Figure 3, along with ABI imagery and local storm reports. You can also interrogate time series for this storm, saved here. 

Figure 1: ProbSevere contours (outer contour is for ProbTor value), MRMS MergedRef, and NWS severe weather warnings for a supercell in central Kansas.  

Figure 2: Time series of PSv3 models for the storm in Figure 1. 

Figure 3: ICP contours and local storm reports for a storm in central Kansas.

Even though the big storms on the Plains usually get all of the attention, severe weather was ongoing elsewhere. In Akron, Ohio, for instance, a storm in a more marginal environment (30 kt eff. shear; 700 J/kg MUCAPE) downed numerous trees. At 17:06 UTC, about 30 minutes before the first reports of downed trees, PSv3 was at 62% while PSv2 was 24%. The strong mean wind 1-3 km AGL (33 kts), moderate ENI lightning density (0.66 fl/km^2/min), favorable 0-3 km lapse rate (8.2 C/km) and MRMS azimuthal shears were the highest contributors to PSv3 at this time. 

Figure 4: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings for a storm near Akron, Ohio.

Figure 5: Time series of PSv3 and PSv2 for the storm in Figure 4. 

And yesterday, a storm in far southern Illinois damaged mobile homes near Vienna. PSv3 was picking up on this storm much better than PSv2, with a probability of 46% about 15 minutes before the report (PSv2 was 3%). The MRMS VIL (27 kg/m^2), low-level lapse rate (7.9 C/km), ENI lightning density (0.34 fl/km^2/min), MRMS 3-6 km azimuthal shear (moderate), and satellite growth rate (moderate) were the top contributors. 

We hope these examples illustrate some of the improvements users can expect to see with PSv3.

Figure 6: ProbSevere contours and MRMS MergedRef for a storm in southern Illinois.

Figure 7: Time series of PSv3 and PSv2 for the storm in Figure 6.

A note on ProbSevere calibration

ProbSevere v3 (PSv3) models are gradient-boosted decision tree classifiers, which generally produce better calibration of probabilities (i.e., the probability values better match the frequency of reports) than the naive Bayesian classifiers of ProbSevere v2 (PSv2). So, forecasters will aptly observe lower probabilities in PSv3, in general. 

The models are trained, validated, and calibrated against NCEI Storm Data reports. Reports from this database are matched up with ProbSevere objects, representing the "truth" or "labels" of the dataset. It is well-known that Storm Data has reporting biases and artifacts, but is still generally regarded as the best nationwide severe-weather-reporting database. What this all means is that while PSv3 models are very well calibrated to Storm Data reports, they may underforecast actual severe weather occurrence in some cases (this is because Storm Data reports are only a subset of all actual severe weather). 

We have seen underforecasting occur in some hail-producing storms. Here is an example in northern Texas. A supercell produced numerous hail reports (up to 3" in diameter). PSv3 topped out at about 80%, whereas PSv2 was > 95%. This was a no-doubt-about-it hailer, with MRMS MESH exceeding 3" briefly (Figures 1 and 2). 

Figure 1: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings (red and yellow polygons) for a storm near Spearman, Texas.

Figure 2: Time series os PSv3 and PSv2 for the storm in Figure 1. 

Here is another example on the Kansas / Colorado line. This storm was warned for several hours, achieved a maximum MRMS MESH of 1.8", yet never resulted in any reports (based on the SPC log). Though the population density is low in this region, severe hail was reported on storms just to the north and east of this storm. PSv3 was generally 20-30% less than PSv2 throughout the storm's history. It's certainly possible that a storm like this actually produced severe hail, but it simply went unreported. If that was the case, storms like this could dilute the severe class during training of the models, affecting model calibration. 

Figure 3: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings (red and yellow polygons) for a storm near Coolidge, Kansas.

Figure 4: Time series os PSv3 and PSv2 for the storm in Figure 3.

So, practically speaking, forecasters should expect lower probabilities for PSv3 compared to PSv2, and mental "warning thresholds" may need to be adjusted (e.g., "30% is the new 50%"; "60% is the new 80%"). The improved calibration also resulted in more skillful models, not just lower probabilities; there are many fewer false alarms and a number of examples where PSv3 is correctly 20-40% greater than PSv2. We hope this helps users to understand ProbSevere's calibration better and ultimately aid in its utility during warning operations. 

Long-lived storm in Illinois

A long-lived, periodically tornadic storm churned through Illinois yesterday. The storm formed ahead of an approaching cold front and produced large hail and severe straight-line wind gusts along with tornadoes. Tornadoes touched down west and east of Springfield, Illinois.

ProbTor v3 (PTv3) demonstrated a better handle on the tornadic threat than ProbTor v2 (PTv2). PTv3 uses gradient-boosted decision trees to predict the probability of a tornado in the near-term for a given storm. It incorporates more NWP, MRMS, and GOES-16 fields than PTv2. 

A predictor analysis at 21:40 UTC showed that the MRMS 0-2 km and 3-6 km azimuthal shears were highly contributing to the probability, which was 34% at this time (PTv2 was 6%). The significant tornado parameter [effective layer] (STP) was also helping, with a value of 0.8. The effective storm-relative helicity (ESRH) was about 140 J/kg, whereas the 0-1 km SRH was only 85 J/kg, which could be one reason PTv2 was rather low, as PTv2 does not use the ESRH as a predictor. 

At the HWT in 2021, forecasters will also be able to see any given storm's ProbSevere history with the a double-click of the storm object in AWIPSII. We hope this will improve users' situational awareness and help with warning-decision making. 

Figure 1: ProbSevere (colored contours) and MRMS MergedRef for a storm in central Illinois. Inset shows the history of the storm's ProbSevere probabilities. 

Typically, PTv3 is lower than PTv2 values, as a result of improved probability calibration in PTv3 (e.g., PTv2 was sometimes "too hot" and over-predicted the threat). But in this case, PTv3 better highlighted the tornadic threat with greater probabilities than v2.

Figure 2: Time series of ProbSevere v3 models. 

Windy in Philly

A modest-looking thunderstorm downed numerous trees and power lines in the Philadelphia area yesterday evening. This storm resided in a high-shear low-CAPE environment (50 kt and 500 J/kg, respectively). 

Figure 1: ProbSevere (storm contours), MRMS MergedRef, and NWS severe weather warnings (yellow polygons) for a storm in the Philadelphia area. 

ProbSevere v3 (PSv3), was able to get a better handle on this storm than v2. PSv3 increased to 28% at 23:40 UTC, the time of the first report. This increase was due to a favorable environment (eff. bulk shear = 53 kt; Meanwind 1-3 km AGL = 41 kt; STP = 0.6, 0-3 km lapse rate = 6.8 C/km) and increasing lightning density (though the storm still had a low flash rate, overall). 

At 23:54 UTC, PSv3 was 41%, with MRMS azimuthal shears increasing modestly. Though the probabilities were somewhat low, PSv3 showed a good improvement over PSv2, which gave probabilities of < 10% almost throughout. 

Forecasters will be able to evaluate PSv3 this spring at the HWT. An offline analysis of PSv3 has found it better calibrated and more skillful that v2, overall. 

Figure 2: Time series of PSv3, PSv2, NWS warnings, and severe local storm reports. 

ProbSevere time series tool

At the 2021 HWT, forecasters will be able to use a new feature of the ProbSevere AWIPS plug-in: a time series tool. Forecasters from previous HWTs have consistently given positive feedback on a web-based meteogram tool, and so we have implemented something similar in AWIPS. 

To use it, you simply double-click on a ProbSevere object, and a window opens up with the time series of ProbHail, ProbWind, ProbTor, and ProbSevere (prob. of any hazard) for the given storm. We hope this will help forecasters better monitor the trends in hazard probabilities.

Figure 1: A severe-hail-producing storm in northeastern North Carolina, and the associated history of its ProbSevere probabilities in a time series window (ProbSevere was equal to ProbHail for this storm).

ProbSevere v3 will also be demonstrated at this year's HWT. PSv3 is driven by a new statistical model (gradient-boosted decision trees) and incorporates new MRMS, ABI, GLM, and SPC mesoanalysis data. This storm was warned at 17:31 UTC, and produced 1-inch hail at 17:38 UTC. At 17:26 UTC, ProbHail v3 jumped to 25%, whereas v2 was only 4%. ProbHail v3 might have been able to highlight this strengthening storm to the forecaster, whereas version 2 did not. A predictor importance analysis of ProbHail v3 for this storm at 17:26 UTC revealed that the highest contributing predictors were:

1. MRMS reflectivity at -20C (52 dBZ)

2. Eff. bulk shear (40 kt)

3. MRMS composite reflectivity (66 dBZ)

4. MRMS MESH (0.55 in)

5. Wet-bulb 0C height (7180 ft)

We expect that ProbSevere v3 will be more accurate and better calibrated than ProbSevere v2, meaning the probabilities more closely match severe report occurrence. 

Windstorm in east Texas

Diffluent flow at 250 mb and a strong theta-E gradient at 850 mb helped spawn a lone severe storm in east Texas last night. This elevated and fast-moving storm caused multiple reports of trees and power lines down in Jasper and Newton counties around 08:00 UTC. 

ProbSevere v3 (PSv3) indicated increasing intensity in this storm about 12-15 minutes before v2. PSv3 uses a different machine-learning model (gradient-boosted decision trees) and utilizes more ABI, GLM, and MRMS data. It also leverages some SPC mesoanalysis fields. UW-CIMSS is running PSv3 experimentally in near-realtime, and it will be evaluated at the 2021 HWT. 

While we've found that PSv3 should be more skillful and better calibrated (i.e., the probabilities better match report occurrence frequency), it may be more difficult to attribute changes in the model probabilities to changes in its predictors. We are working on methods to convey predictor importance to users on a per-storm basis in near-realtime. 

Post-mortem, we evaluated the predictor importance for this storm at 07:34 UTC. We found the most important contributors were:

1. MRMS composite reflectivity (64 dBZ)
2. Eff. bulk shear (58 kt)
3. ENI total lightning density (0.29 flashes/min/km^2)
4. MRMS 0-2 km  azimuthal shear (0.011 s^-1)
5. MRMS VIL (23 g / m^2)
6. MRMS 3-6 km azimuthal shear (0.006 s^-1)
7. ABI + GLM intense convection probability (95%)

The 0-3 km lapse rate (3.7 C/km) was the predictor detracting from the probability the most. 

We hope that PSv3 will give users even more confidence during severe weather warning operations. 

Nocturnal hailstorms in the Upper Midwest and ProbSevere v3

A cold front spawned numerous hail-producing storms in the Upper Midwest during the evening and overnight hours last night. ProbSevere version 3 (PSv3), which uses gradient-boosted decision trees with MRMS, ABI, GLM, Earth Networks, and NWP/SPC mesoanalysis data, was able to improve guidance over version 2 (PSv2) for a number of storms.

One of the more prolific hail-producing storms of the day formed west of Minneapolis, MN. PSv3 improved upon PSv2 with higher probabilities at a critical time before the first NWS warning was issued. 

Figure 1: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings for a storm west of Minneapolis, MN.

Figure 2: Time series of PSv3 and PSv2 for a select time period for the highlighted storm in Figure 1.

Looking at 01:50 UTC from the time series in Figure 2, we found that the top five predictors aiding to higher severe probabilities were:

1. ENI max total lightning density (0.28 flashes/min/km^2)
2. Eff. bulk shear (41 kt)
3. MRMS max 3-6 km azshear (0.008 /s)
4. MRMS max composite reflectivity (60.5 dBZ)
5. ABI+GLM intense convection probability (98%)

The intense convection probability (ICP) is an ABI + GLM based deep-learning model used to diagnose intense convection, and is a predictor in PSv3. 

Another storm to the northeast of the storm above also briefly produced severe hail. While the PSv3 probability was relatively low, it was still much higher than PSv2, which is an improvement that may be enough to bring a forecaster's attention to the storm.

Figure 3: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings for a storm WNW of Minneapolis, MN.

Figure 4: Time series of PSv3 and PSv2 for a select time period for the highlighted storm in Figure 2.

PSv3 peaked at 37% at 02:08 UTC, 6 minutes before the hail report. Performing a predictor importance analysis at this time, we found that the top five predictors aiding to higher severe probabilities were:

1. MRMS max composite reflectivity (65.5 dBZ)
2. Eff. bulk shear (47 kt)
3. MRMS max VIL (25 g/m^2)
4. MRMS max 3-6 azshear (0.007 /s)
5. MRMS max MESH (0.56 in)

A storm in northwestern Wisconsin fluctuated in probability, but eventually produced 1.25-inch hail.

At the time of the report (03:00 UTC), PSv3 was 47% and PSv3 was 13%. 

Figure 5: ProbSevere contours and MRMS MergedRef for a hail-producing storm in northwest Wisconsin.

At the time of the report, we found the top five predictors in PSv3 were:

1. MRMS max MESH (0.76 in)
2. MRMS max VIL (31 g/m^2)
3. Eff. bulk shear (52 kt)
4. MRMS max composite reflectivity (61.5 dBZ)
5. ENI total lightning density (0.21 flashes/min/km^2)

PSv3 should also help reduce false alarms around the country, relative to PSv2. This warned storm in northern Nebraska serves as an example. The maximum probability for PSv3 was 17%, compared to 49% for PSv2. There were no reports recorded for this storm.

Figure 6: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings for a storm in northern Nebraska.

One more example, just west of Duluth, MN, where PSv3 was about 20% higher than PSv2 at the time of a 1.25-inch hail report. At the time of the report, we found the top five predictors in PSv3 were:

1. MRMS max MESH (1.19 in)
2. MRMS max composite reflectivity (67.5 dBZ)
3. MRMS max VIL (26 g/m^2)
4. ENI total lightning density (0.23 flashes/min/km^2)
5. MRMS max 3-6 km azshear (0.007 /s)

Figure 7: ProbSevere contours and MRMS MergedRef for a storm west of Duluth, MN.

We hope these examples illustrate the improvement that forecasters will experience in using PSv3 at the Hazardous Weather Testbed. We also hope that they illustrate the benefit of intelligently fusing data together from the different observation systems we have available. We are actively working on ways to help users better interpret model predictions using analyses such as predictor importance ranking.

ProbSevere v3 in Florida

A sagging cold front provided a marginal risk of severe weather in south Florida. The experimental ProbSevere v3 (PSv3) will be evaluated at the Hazardous Weather Testbed this spring and summer. PSv3 models use a different machine-learning method, and incorporate additional MRMS, ABI, GLM, and SPC mesoanalysis fields. We've found that the models should be more skillful and better calibrated, overall.

This storm caused damage to silos and chicken barns just north of Lake Okeechobee yesterday afternoon. PSv3 showed much higher probabilities (≥ 40%), whereas PSv2 maxed out at 9% before the wind report. PSv3 should provide better guidance on severe storms for both busy severe days and marginal severe days.

By inspecting the predictor importance of this storm right before the wind report, it was found that the top-5 contributing predictors were:

1. ENI total lightning density (0.45 fl/km^2/min)
2. ABI satellite growth rate (3.8 %/min)
3. MRMS VIL (29 g/m^2)
4. Eff. shear (42 kt)
5. 0-3 km lapse rate (7.8 C/km)

We are currently working on how best to convey predictor importance to forecasters in AWIPS, which we hope will help users better understand why the model makes the predictions that it does, and ultimately better utilize ProbSevere guidance.

Nocturnal tornado in Arkansas

 A strong, upper-level trough ejected from the Southern Plains toward the Mississippi Valley on Saturday night, carrying with it severe storms along and ahead of a cold front in Arkansas, fueled by low-level moisture and a very strong mid-level jet streak (75 - 90 kt). A portion of the squall line quickly increased in the probability of tornado, indicated by the ProbTor model (Figure 1).

Fig. 1: ProbTor contour north of Little Rock, AR, 66% probability of tornado; MRMS MergedReflectivity (shaded).

Soon after the timestamp in Figure 1, this storm dropped an EF-1 tornado in the small town of Romance, AR, which destroyed or damaged numerous homes and resulted in at least 4 injuries. The NWS in Little Rock, AR noted that there was a brief but concentrated area of rotation as well as a debris signature. 

Figure 3 demonstrates the large increase in 0-2 km azimuthal shear at around 06:40 UTC, coupled with a very conducive environment for tornadoes (eff. shear ≥ 50 kt, 1-3 km AGL mean wind ≥ 60 kts, 0-1 km storm-relative helicity ~ 400 J/kg) led to the rapid increase in ProbTor. There was very little lightning activity evident with this storm.

With MRMS v12 now in operations, NWS forecasters can receive ProbSevere output in their offices. There was also an update to the azimuthal shear products in MRMS v12, improving their accuracy and reducing false alarms. This should help reduce the false alarms of algorithms dependent on MRMS azimuthal shear, such as ProbTor.

Fig. 2: Time series of the ProbSevere models as well as local severe storm reports and NWS severe weather warnings.

Fig. 3: Time series of the ProbTor model, select constituent predictors, as well as local severe storm reports and NWS severe weather warnings.