Automated Procedure to Assess Civil Infrastructure Data Quality: Method and Validation

Abstract

Monitoring data are collected to measure the condition, environment, usage, and performance of civil infrastructure. High quality monitoring data are necessary for decision-support systems, design analysis, and research. However, little work has been done in the area of generic, automated data quality assessment and cleansing procedures. We have developed an automated, two-level data quality assessment procedure to address this deficiency. In the first level of our procedure, several different data quality assessment methods are used in a voting scheme to identify concentrations of anomalies in aggregate data. In the second level, differences between anomalies and normal data at the individual data level are identified; combined with domain knowledge, these differences can be used to identify different types of errors, such as missing data and calibration errors. In our case studies, we have been able to effectively cleanse the data using the results from our data quality assessment procedure. We have also developed a test bench to explore the sensitivity of the data quality assessment algorithms used in our approach. The test bench introduces a known error into a clean, artificial data set and then evaluates how well each assessment method identifies the error. The test bench results show that our approach is able to effectively identify anomalies, even those with small magnitudes of error.