Evaluating Rank Histograms Using Decompositions of the Chi-Square Test Statistic

Abstract

Rank histograms are often plotted to evaluate the forecasts produced by an ensemble forecasting system?an ideal rank histogram is ?flat? or uniform. It has been noted previously that the obvious test of ?flatness,? the well-known ?2 goodness-of-fit test, spreads its power thinly and hence is not good at detecting specific alternatives to flatness, such as bias or over- or underdispersion. Members of the Cramér?von Mises family of tests do much better in this respect. An alternative to using the Cramér?von Mises family is to decompose the ?2 test statistic into components that correspond to specific alternatives. This approach is described in the present paper. It is arguably easier to use and more flexible than the Cramér?von Mises family of tests, and does at least as well as it in detecting alternatives corresponding to bias and over- or underdispersion.