When Threshold and Metric Selection Matter for Resilience Planning in an Uncertain and Changing World

Abstract

Translation of a system’s performance goals into measurable quantities is fundamental for water systems planning. Frequently, the selected metrics are the same threshold-based metrics that have been common in water systems analysis for four decades. These are single statistic measures that characterize failure characteristics (e.g., frequency, duration, and magnitude) for a given simulation period where the failure threshold is a target delivery. At the same time, our systems are increasingly challenged by shifting water availability due to climate change and other anthropogenic pressures. Preparing our water systems to be resilient to such changes is a major challenge faced by water resources planners and managers. In particular, shifting mismatches between availability and target deliveries mean that for optimal design and control problems, metric choice implicitly becomes a statement of preference of failure type (i.e., the magnitude and/or duration of failure events). This usually occurs without explicit discussion of failure preference, resulting in unintended consequences that are not well understood. This study addresses the issue of unintended consequences in two ways. First, we introduce an approach to detect the threshold at which the balance between availability and delivery is no longer stable. Second, we characterize the consequences of metric choice when this threshold is exceeded. We find that early warning signals, and specifically the L moment estimator of variance, are able to detect the system’s entry into a state of increasing failure. We also find that for target deliveries greater than this threshold there are increasing tradeoffs between failure frequency, duration, and magnitude and that the distributional characteristics of failure magnitude and duration are driven by metric selection, objective formulation, and where on the Pareto front the solution sets lies. This study establishes a way to detect critical system thresholds and characterizes the effects of metric and threshold selection on failure, both of which are requisite for analysts and decision makers to avoid unintended consequences of planning and management decisions in an uncertain and changing world. Metrics form the basis of comparison for water resources planning and management decisions. When availability and demand are mismatched, the use of the standard threshold-based metrics in optimization problems can easily lead to unintended consequences. This is of fundamental concern for water resources planning and management in a changing world. This study helps analysts and decision makers avoid unintended consequence of poor metric selection by (1) characterizing the influence of metric selection on the types of failure events the system may endure; and (2) introducing a measure to detect the threshold at which analysts and decision makers should begin to be cautious of such consequences.