Fusing Data Extracted from Bridge Inspection Reports for Enhanced Data-Driven Bridge Deterioration Prediction: A Hybrid Data Fusion Method

Abstract

Data buried in textual bridge inspection reports offer great promise for enhanced data-driven bridge deterioration prediction. However, learning from these reports is challenging because they typically use multiple concept names to refer to the same entity and typically describe multiple instances of the same type of deficiency. Such multiple names and instances increase the dimensionality and the sparsity of the feature space, which would cause overfitting to a particular feature, undermine the generalizability of the machine learning models, and compromise the performance of the data-driven prediction. To address this challenge, this paper proposes a new hybrid data fusion method. It combines an unsupervised named entity normalization method and an entropy-based numerical data fusion method to fuse concept names and numerical data, respectively. The proposed normalization method uses an n-gram model to generate candidate canonical identifier names and utilizes corpus statistics and lexical patterns to fuse the multiple concept names into a candidate name that balances abstraction and detailedness. The proposed fusion method uses data discretization and information entropy to fuse the multiple deficiency measures (of the instances) into a single representation. The hybrid fusion method was validated in fusing data extracted from textual bridge inspection reports for supporting the prediction of future bridge condition ratings. Learning from the fused data, compared to learning from the unfused data, improved the accuracies of predicting the ratings of decks, superstructures, and substructures by 8.0%, 8.5%, and 7.9%, respectively.