A Case Study of Deterministic Forecast Verification: Tropical Cyclone Intensity

Abstract

Deterministic predictions of tropical cyclone (TC) intensity from operational forecast systems traditionally have been verified with a summary accuracy measure (e.g., mean absolute error). Since the forecast system development process is coupled to the verification procedure, it follows that TC intensity forecast systems have been developed with the goal of producing predictions that optimize the chosen summary accuracy measure. Here, the consequences of this development process for the quality of the resultant forecasts are diagnosed through a distributions-oriented (DO) verification of operational TC intensity forecasts. DO verification techniques examine the full relationship between a set of forecasts and the corresponding set of observations (i.e., forecast quality), rather than just the accuracy attribute of that relationship. The DO verification results reveal similar first-order characteristics in the quality of predictions from four TC intensity forecast systems. These characteristics are shown to be consistent with the theoretical response of a forecast system to the imposed goal of summary accuracy measure optimization: production of forecasts that asymptote with lead time to the central tendency of the observed distribution. While such forecasts perform well with respect to the accuracy, unconditional bias, and type I conditional bias attributes of forecast quality, they perform poorly with respect to type II conditional bias. Thus, it is clear that optimization of forecast accuracy is not equivalent to optimization of forecast quality. Ultimately, developers of deterministic forecast systems must take care to employ a verification procedure that promotes good performance with respect to the most desired attributes of forecast quality.