Accuracy and Potential Economic Value of Categorical and Probabilistic Forecasts of Discrete Events

Abstract

The accuracy and the potential economic value of categorical and probabilistic forecasts of discrete events are discussed. Accuracy is assessed applying known measures of forecast accuracy, and the potential economic value is measured by a weighted difference between the system probability of detection and the probability of false detection, with weights function of the cost?loss ratio and the observed ratio and the observed relative frequency of the event. Results obtained using synthetic forecast and observed fields document the sensitivity of accuracy measures and of the potential forecast economic value to imposed random and systematic errors. It is shown that forecast skill cannot be defined per se but depends on the measure used to assess it: forecasts judged to be skillful according to one measure can show no skill according to another measures. More generally, it is concluded that the design of a forecasting system should follow the definition of its purposes, and should be such that the ensemble system maximizes its performance as assessed by the accuracy measures that best quantify the achievement of its purposes. Results also indicate that independently from the model error (random or systematic) ensemble-based probabilistic forecasts exhibit higher potential economic values than categorical forecasts.