User-Oriented Two-Dimensional Measure of Effectiveness for the Evaluation of Transport and Dispersion Models

Abstract

A two-dimensional measure of effectiveness for comparing hazardous material transport and dispersion model predictions and field observations has been developed. This measure is used for comparing predictions and observations paired in space and time, and the components of this measure?overprediction, or false-positive fraction, and underprediction, or false-negative fraction?can illuminate strengths and weaknesses of a model in ways that many one-dimensional measures cannot. Comparisons of predictions of short-range field observations are used to illustrate features of this measure of effectiveness, including its computation based on dosages or based on a dosage threshold of interest. With this user-oriented measure of effectiveness, statistically significant resolution of transport and dispersion model performance differences as a function of downwind range and meteorological stability category grouping, as well as between different models, is described. Evaluation of probabilistic prediction outputs is also demonstrated with this measure of effectiveness. The user-oriented measure of effectiveness can be evaluated for different assumed hazardous materials and human effects of interest and, as such, can relate the quality of a prediction for agents of greatly varying toxicity. A quantitative method that can aid in the communication of a user's risk tolerance is identified and allows one to divide the two-dimensional measure of effectiveness space into regions of relative acceptability for the particular user and application.