On the Failure of the Area Metric for Validation Exercises of Stochastic Simulations

Abstract

This paper discusses the application of the area metric to the quantification of modeling errors. The focus of the discussion is the effect of the shape of the two distributions on the result produced by the Area Metric. Two different examples that assume negligible experimental and numerical errors are presented: the first case has experimental and simulated quantities of interest defined by normal distributions that require the definition of a mean value and a standard deviation; the second example is taken from the V&V10.1 ASME standard that applies the Area Metric to quantify the modeling error of the tip deflection of a loaded hollow tapered cantilever beam simulated with the static Bernoulli–Euler beam theory. The first example shows that relatively small differences between the mean values are sufficient for the area metric to be insensitive to the standard deviation. Furthermore, the example of the V&V10.1 ASME standard produces an Area Metric equal to the difference between the mean values of experiments and simulations. Therefore, the error quantification is reduced to a single number that is obtained from a simple difference of two mean values. This means that the area metric fails to reflect a dependence on the difference in the shape of the distributions representing variability. The paper also presents an alternative version of the Area Metric that avoids this filtering effect of the shape of the distributions by utilizing a reference simulation that has the same mean value as the experiments. This means that the quantification of the modeling error will have contributions from the difference in mean values and the shape of the distributions.