Evaluating Operational Risk in Environmental Modeling: Assessment of Reliability and Sharpness for Ensemble Selection

Abstract

To adequately define risk in an operational setting, modeling and data uncertainty must be addressed. Though metrics to evaluate model performance are numerous in the literature, few integrate either modeling uncertainty or benchmark data uncertainty, and even fewer integrate both. The Combined Overlap Percentage (COP) ensemble metric is a notable exception: it is based on optimizing the trade-off objectives of maximizing the overlap between simulated and benchmark uncertainty bounds (overlap-reliability) while minimizing simulated ensemble uncertainty bound width (overlap-sharpness) with equal weight. We further develop the COP by assessing weighting methods to increase applicability to additional types of benchmark data uncertainty. As new advanced datasets are generated each year, the weighted COP can integrate ensembles of benchmark data rather than forcing modelers to attempt to identify the best product at a low computational cost. The new weighting method further allows the COP to adapt to the unique features of those new datasets. Results suggest increasing the weight of overlap-sharpness when robust benchmark uncertainty estimates are available. Conversely, higher weights should be given to overlap-reliability when little benchmark uncertainty information is available. Finally, timestep weighting and data transforms are only impactful if overlap-sharpness is prioritized. The results are particularly relevant in an operational context and could allow for the integration of uncertainty into calibration and ensemble generation at a low computational cost. Uncertainty represents the expected range a perfect measurement would span if it were possible to collect a perfect measurement. In modeling applications, this range expands to include the imperfections of the model. Recognizing this, uncertainty in modeling applications has generally led to the creation of over-confident models fit to, or evaluated against, imperfect data that are assumed to be perfect. Including uncertainty bounds representing a range of observations when evaluating a model is not often considered despite many studies highlighting its importance; this is referred to as output uncertainty. Here the authors further develop the Combined Overlap Percentage (COP) with new weighting factors, which allows for the integration of observation, or benchmark, data uncertainty in an accessible and computationally reasonable way. New weighting schemes allow the updated COP to capture periods of low data uncertainty, which allows for higher quality data (that for which there is higher confidence in) to have a greater influence on the model. The authors test the weighted COP with three case studies to explore how much information is needed to apply this method, and how weighting schemes perform with different amounts of uncertainty information. Finally, the authors make recommendations based on the results generated by the case studies.