Protocol for the Validation of Models for Regional Risk Analysis

Abstract

Regional risk analysis provides information for decisions made by communities, state and federal agencies, and the insurance industry. The analyses involve comprehensive prediction models, including nested models in complex multistep procedures. While numerous models are available, they are often not validated due to limited data availability and measurement challenges. However, validation is crucial since inaccurate predictions may result in suboptimal decisions. Thus, this paper proposes three measures to validate the predictive ability of models used in regional risk analysis (i.e., the Accuracy Likelihood, Prediction Error, and Distribution Match). The Accuracy Likelihood quantifies the probability of observing the recorded data under the predictive model’s hypotheses/assumptions. The Prediction Error measures the difference between the recorded value and values predicted by a model. The Distribution Match measures the similarity between the probability distributions of the predicted quantities and the corresponding empirical distributions of the recorded data. As an example, we check the predictive validity of seismic risk analysis models using data from the 2016 Kumamoto earthquake in Mashiki City, Kumamoto, Japan. We consider three sets of models [i.e., from HAZUS, MAEViz, and local Kumamoto Prefecture Models (KPM)] to predict the ground motion intensity, and physical damage on buildings, bridges, electric power infrastructure, and potable water and wastewater infrastructure. The comparison shows the predictive power of some of the available models and drives future research toward essential enhancements.