A Note On the Maximum Peirce Skill Score

Abstract

Binary classifiers are obtained from a continuous predictor using a threshold to dichotomize the predictor value into event occurrence and nonoccurrence classes. A contingency table is associated with each threshold, and from this table many statistical indices (like skill scores) can be computed. This work shows that the threshold that maximizes one of these indices [the Peirce skill score (PSS)] has some important properties. In particular, at that threshold the ratio of the two likelihood distributions is always 1 and the event posterior probability is equal to the event prior probability. These properties, together with the consideration that the maximum PSS is the point with the ?most skill? on the relative operating characteristic curve and the point that maximizes the forecast value, suggest the use of the maximum PSS as a good scalar measure of the classifier skill. To show that this most skilled point is not always the best one for all the users, a simple economic cost model is presented.