Training Neural Networks by Adaptive Random Search Techniques

Abstract

A relatively simple stochastic optimization procedure based on the adaptive random search algorithm is presented to train artificial neural networks of the type encountered in applied mechanics applications. After discussing some essential features of the algorithm that influence its search efficiency, a procedure is outlined for replacing the back-propagation training approach by the new method in order to train networks involving high-dimensional parameter vectors. The method is successfully used in conjunction with a multilayer network involving a parameter vector of very high dimension. It is shown that the adaptive random search approach shifts the training effort from the user to the computer by exchanging additional computer search effort for easier training tasks on the part of the user. Extensive simulation studies are presented to provide statistically significant results related to the characteristics of the stochastic training approach. Guidelines are provided for applying the method to generic neural network training episodes.