Comments on “Strategies for Assessing Skill and Significance of Screening Regression Models with Emphasis on Monte Carlo Techniques”

Abstract

In a recent paper, Lanzante reviewed methods for estimating the skill and significance of screening regression models through the use of Monte Carlo simulations. The strategies reviewed have several limitations that were not specified by the author. Due to the influence of true model skill, the Monte Carlo method provides an upper bound on the expected artificial skill, not the expected artificial skill itself as assumed. Lanzante emphasizes the advantages of the use of independent (uncorrelated) predictors. However, the disadvantages of their use and the advantages of dependent predictors in a screening regression were not considered. The review of the effects of serial correlation on estimates of skill is misleading. The assertion that the formulations developed by Davis and Chelton are erroneous is incorrect. Moreover, contrary to the implication of the review, the use of effective sample size in tests of model significance has practical utility in applications including the Monte Carlo method.