Comparison of Objective and Subjective Precipitation Probability Forecasts: The Sufficiency Relation

Abstract

In this paper the sufficiency relation is used to compare objective and subjective probability of precipitation (PoP) forecasts. The theoretical significance of the sufficiency relation in comparative evaluation arises from the fact that if it can be shown that forecasting system A is sufficient for forecasting system B, then A's forecasts are necessarily of higher quality and greater value to all users than B's forecasts. However, since the sufficiency relation is an incomplete order (it is not always possible to show that system A is sufficient for system B, or vice versa), the practical significance of this relation warrants further investigation. An operational method of comparing forecasting systems using the sufficiency relation has recently been described in the forecasting literature. This method involves the construction of a so-called forecast sufficiency characteristic (FSC) for each forecasting system, based on a representative set of forecasts and observations. In terms of this characterization, system A is sufficient for system B if A's FSC is superior to B's FSC. Objective and subjective PoP forecasts for six National Weather Service offices are compared here in terms of their respective FSCs. Sufficiency was found in only two of the 24 cases defined by various combinations of forecast office, season, and lead time. In these two cases, both involving 12?24 hour forecasts, the subjective forecasts were sufficient for the objective forecasts. Several other cases exhibited a condition described as "almost sufficient,? but caution must be exercised in drawing conclusions regarding the relative quality and/or relative value of forecasts in such cases. Comparison of the FSCs of PoP forecasts and the FSCs of categorical forecasts derived from the PoP forecasts reveals that, as expected, the former are always superior to the latter. The implications of these results for comparative evaluation of weather forecasting systems are discussed and some possible extensions of this work are identified.