Domain Randomization Using Deep Neural Networks for Estimating Positions of Bolts

Abstract

Current manual practices of replacing bolts on structures are time-consuming and costly, especially because of numerous bolts. Thus, an automated method that can visually detect and localize bolt positions would be highly beneficial. We demonstrate the use of deep neural networks using domain randomization for detecting and localizing bolts on a workpiece. In contrast to previous approaches that require training on real images, the use of domain randomization enables all training in simulation. The key idea is to create a wide variety of computer-generated synthetic images by varying the texture, color, camera position and orientation, distractor objects, and noise, and train the neural network on these images such that the neural network is robust to scene variability and hence provides accurate results when deployed on real images. Using domain randomization, we train two neural networks, a faster regional convolutional neural network for detecting the bolt and placing a bounding box, and a regression convolutional neural network for estimating the x- and y-position of the bolts relative to the coordinates fixed to the workpiece. Our results indicate that in the best case, we can detect bolts with 85% accuracy and can predict 75% of bolts within 1.27 cm accuracy. The novelty of this work is in using domain randomization to detect and localize: (1) multiples of a single object and (2) small-sized objects (0.6 cm × 2.5 cm).