The Verification of Probability Forecasts

Abstract

Brier's scoring procedure for the evaluation of probability statements is analyzed to show two important aspects of the forecasting process. The first is a sorting process in which the forecaster assigns each prediction to one of a set of ordered classes of likelihood of occurrence of the meteorological event. The second is a labeling process in which he assigns a numerical value to each class. This value is intended to be the relative frequency (or probability) of occurrence of the event for the predictions in that class. When forecasts are evaluated relative to the use of simple statements such as climatological probability, the Brier score is shown to consists of a sorting gain and a bias (or mislabeling) penalty. Evidence is presented to show that meteorological forecasts made by humans have appreciable sorting skill and suffer little bias penalty. The relevance of the bias penalty is attacked and defended.