OTL-PEM: An Optimization-Based Two-Layer Pointwise Ensemble of Surrogate Models

Abstract

The ensemble of surrogate models is increasingly implemented in practice for its more flexibility and resilience than individual models. The optimization-based two-layer pointwise ensemble of surrogate models (OTL-PEM) is proposed in this paper as a novel pointwise ensemble of surrogate models. The framework of two-layer surrogate models is defined in the OTL-PEM, with data-surrogate models having several types of individual surrogate models fitting the given dataset. In contrast, the weight-surrogate models are modeled based on the cross-validation errors aiming to fit the pointwise weights of several individual surrogate models. To avoid the negative influence of the poor individual surrogate models, the model selection problem is transformed into several optimization problems, which can be solved easily using a sophisticated optimization algorithm to eliminate the globally poor surrogate models. In addition, the optimization space is extracted to reduce the prediction instability caused by weight-surrogate model extrapolation. More than 40 test functions are used to select the appropriate hyperparameters of the OTL-PEM and evaluate the OTL-PEM’s performance. The results show that the OTL-PEM can provide more accurate and robust approximation performance than individual surrogate models and other ensembles of surrogate models.