Reliable Benchmarking of Breast Ultrasound Lesion Classification Requires Patient-Level Validation

Abstract

Abstract. Breast ultrasound is widely used for lesion characterization, yet reported deep-learning performance varies substantially with dataset composition, preprocessing, and validation design. A major source of bias arises when patient-level separation is not enforced, allowing correlated images from the same subject to inflate performance estimates. This study presents a leakage-aware benchmark of convolutional neural networks (CNNs), a Vision Transformer (ViT), and a CNN–transformer late-fusion configuration for benign-versus-malignant breast ultrasound classification on the BUS-UCLM dataset. After exclusion of normal-category images, the final cohort comprised 264 images from 36 patients, including 174 benign and 90 malignant images. Seven CNN-family models, one ViT baseline, and one ResNet18–ViT probability-level late-fusion configuration were evaluated using strict patient-level fivefold cross-validation. Additional analyses included fusion ablation, paired Wilcoxon signed-rank testing, gradient-weighted class activation mapping (Grad-CAM) visualization, and an image-level leakage demonstration. Under strict patient-level evaluation, performance was moderate across all models. GoogLeNet achieved the highest mean accuracy (61.48%), InceptionV3 achieved the highest mean macro-F1 (59.24%), and ResNet50 achieved the highest mean area under the receiver operating characteristic curve (AUC) (0.6699), whereas the standalone ViT showed weaker overall discrimination. The late-fusion configuration remained competitive in threshold-dependent metrics but did not surpass the strongest CNN baselines in AUC. Overall, no architecture demonstrated a clear advantage across both threshold-dependent and threshold-independent metrics. By contrast, image-level splitting substantially inflated apparent performance, underscoring the importance of rigorous patient-level separation for credible benchmarking in breast ultrasound classification.