Semisupervised Clustering Approach for Pipe Failure Prediction with Imbalanced Data Set

Abstract

In recent years, machine learning (ML) approaches have been used widely for water pipe condition assessment and failure prediction. These methods require a considerable amount of data from water distribution networks (WDNs). Imbalanced and missing data, whether asset or failure data, compromise a model’s prediction performance. In this research, using only 2 years of failure data in a real WDN, three ML methods—XGBoost, random forest and logistic regression—were used to prioritize asset rehabilitation. To address the issue of imbalanced data, a novel method of semisupervised clustering is proposed to leverage the domain knowledge in combination with unsupervised learning to divide the data set into homogeneous categories and enhance the classification accuracy. The introduced approach performed better than well-known data science class imbalance treatment techniques. Furthermore, analysis of the results indicated that classification evaluation metrics struggled to assess practically the effectiveness of various methods. To address this, an economic indicator is proposed to rank the pipes for rehabilitation based on their cost and likelihood of failure (LoF). Preventive maintenance using the results of an economic indicator reduces the number of failures with a small fraction of the total replacement cost. Moreover, another indicator was developed to consider the consequence of the failures and LoF simultaneously. This indicator mitigates in a cost-effective manner the flow capacity reductions in WDNs caused by failures. The results of this study provide asset managers with a powerful tool to prioritize assets for rehabilitation. In recent years, machine learning algorithms have gained popularity for assessing water pipe conditions and predicting failures. However, their effectiveness relies on substantial data from water distribution networks (WDNs). Challenges arise with limited (imbalanced) data, affecting prediction accuracy. This study focused on a specific WDN with only 2 years of failure data, aiming to identify priority assets for rehabilitation. Three ML methods (XGBoost, random forest, and logistic regression) and a novel semisupervised clustering approach were employed. This method combines expert knowledge with traditional techniques, significantly improving predictive accuracy. By applying ML algorithms within these homogenous clusters, predictive accuracy was enhanced notably. Two novel metrics were introduced for prioritizing pipe rehabilitation: one combining failure likelihood and replacement costs, and the other evaluating pipes based on their significance within the WDN and associated rehabilitation expenses. These models empower asset managers to optimize pipe replacement budget allocation and enhance the network performance.