On Variability due to Local Minima and K-Fold Cross Validation

Abstract

Resampling methods such as cross validation or bootstrap are often employed to estimate the uncertainty in a loss function due to sampling variability, usually for the purpose of model selection. In models that require nonlinear optimization, however, the existence of local minima in the loss function landscape introduces an additional source of variability that is confounded with sampling variability. In other words, some portion of the variability in the loss function across different resamples is due to local minima. Given that statistically sound model selection is based on an examination of variance, it is important to disentangle these two sources of variability. To that end, a methodology is developed for estimating each, specifically in the context of