Sensitivity of Several Performance Measures to Displacement Error, Bias, and Event Frequency

Abstract

The sensitivity of various accuracy measures to displacement error, bias, and event frequency is analyzed for a simple hypothetical forecasting situation. Each measure is found to be sensitive to displacement error and bias, but probability of detection and threat score do not change as a function of event frequency. On the other hand, equitable threat score, true skill statistic, and odds ratio skill score behaved differently with changing event frequency. A newly devised measure, here called the bias-adjusted threat score, does not change with varying event frequency and is relatively insensitive to bias. Numerous plots are presented to allow users of these accuracy measures to make quantitative estimates of sensitivities that are relevant to their particular application.