A note on ProbSevere calibration

ProbSevere v3 (PSv3) models are gradient-boosted decision tree classifiers, which generally produce better calibration of probabilities (i.e., the probability values better match the frequency of reports) than the naive Bayesian classifiers of ProbSevere v2 (PSv2). So, forecasters will aptly observe lower probabilities in PSv3, in general. 

The models are trained, validated, and calibrated against NCEI Storm Data reports. Reports from this database are matched up with ProbSevere objects, representing the "truth" or "labels" of the dataset. It is well-known that Storm Data has reporting biases and artifacts, but is still generally regarded as the best nationwide severe-weather-reporting database. What this all means is that while PSv3 models are very well calibrated to Storm Data reports, they may underforecast actual severe weather occurrence in some cases (this is because Storm Data reports are only a subset of all actual severe weather). 

We have seen underforecasting occur in some hail-producing storms. Here is an example in northern Texas. A supercell produced numerous hail reports (up to 3" in diameter). PSv3 topped out at about 80%, whereas PSv2 was > 95%. This was a no-doubt-about-it hailer, with MRMS MESH exceeding 3" briefly (Figures 1 and 2). 

Figure 1: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings (red and yellow polygons) for a storm near Spearman, Texas.

Figure 2: Time series os PSv3 and PSv2 for the storm in Figure 1. 

Here is another example on the Kansas / Colorado line. This storm was warned for several hours, achieved a maximum MRMS MESH of 1.8", yet never resulted in any reports (based on the SPC log). Though the population density is low in this region, severe hail was reported on storms just to the north and east of this storm. PSv3 was generally 20-30% less than PSv2 throughout the storm's history. It's certainly possible that a storm like this actually produced severe hail, but it simply went unreported. If that was the case, storms like this could dilute the severe class during training of the models, affecting model calibration. 

Figure 3: ProbSevere contours, MRMS MergedRef, and NWS severe weather warnings (red and yellow polygons) for a storm near Coolidge, Kansas.

Figure 4: Time series os PSv3 and PSv2 for the storm in Figure 3.

So, practically speaking, forecasters should expect lower probabilities for PSv3 compared to PSv2, and mental "warning thresholds" may need to be adjusted (e.g., "30% is the new 50%"; "60% is the new 80%"). The improved calibration also resulted in more skillful models, not just lower probabilities; there are many fewer false alarms and a number of examples where PSv3 is correctly 20-40% greater than PSv2. We hope this helps users to understand ProbSevere's calibration better and ultimately aid in its utility during warning operations.