Evaluation of DenseNet121 decision thresholds and probability calibration on chest radiographs from Mongolian individuals
- VernacularTitle:Монгол хүний цээжний рентген зурагт DenseNet121 загварын шийдвэрийн босго болон магадлалын тохируулгыг үнэлсэн нь
- Author:
Bayasgalant D
1
;
Orgil-Erdene E
2
;
Oyunbold L
3
;
Delgerdalai Kh
4
Author Information
1. Department of Radiology, School of Medicine, MNUMS
2. AI Department, AND Global LLC
3. Department of Radiology, First Central Hospital of Mongolia
4. Department of Radiology, MNUMS
- Publication Type:Journal Article
- Keywords:
sensitivity;
specificity;
decision threshold;
calibration;
external validation
- From:
Mongolian Journal of Health Sciences
2026;97(7):106-110
- CountryMongolia
- Language:Mongolian
-
Abstract:
Background:Mongolia’s vast territory and dispersed population pose challenges to timely diagnostic care. Artificial intelligence may support chest radiograph interpretation and prioritization of suspicious findings for clinician review. Models trained on foreign datasets should therefore be evaluated on Mongolian data before clinical use.
Aim:To externally validate a DenseNet121 chest radiograph AI model trained on the public CheXpert dataset using Mongolian clinical data, and to evaluate local adaptation strategies for clinical translation.
Materials and Methods:A DenseNet121-based multi-label classifier was trained on CheXpert with ImageNet initialization, 320×320 RGB inputs, and 14 sigmoid outputs, including No Finding and Support Devices. The best checkpoint was selected by internal CheXpert validation and evaluated without fine-tuning on 671 Mongolian chest radiographs whose 14 labels were manually extracted by a radiologist from previous clinical reports. Discrimination was assessed using ROC-AUC, macro AUC, and micro AUC. Threshold-dependent performance was evaluated at the default 0.5 threshold and after label-specific Youden index optimization. Youden thresholds were selected and evaluated on the same 671 images, so improvements may be optimistic. Platt calibration used five-fold patient-grouped cross-validation for 12 eligible labels; each patient was evaluated using a calibrator fitted without their data. Brier score and expected calibration error (ECE) were assessed.
Results:The mean 14-label AUC in the internal CheXpert validation set used for model selection was 0.8169. On the Mongolian external validation set, macro AUC was 0.7517 and micro AUC was 0.7762. Strong discrimination was observed for Support Devices (0.9190), No Finding (0.8986), Consolidation (0.8914), Pleural Effusion (0.8793), Edema (0.8425), Pneumothorax (0.8193), and Lung Opacity (0.7997). AUC decreased for Fracture, Atelectasis, Pleural Other, and Cardiomegaly, with the causes of these differences unestablished. At the default 0.5 threshold, macro sensitivity was low (0.2347) despite high macro accuracy (0.8139), indicating that direct transfer may miss positive findings. Local threshold tuning improved F1 scores for Consolidation from 0.006 to 0.794, Edema from 0.509 to 0.600, Lung Opacity from 0.590 to 0.662, and Pleural Effusion from 0.737 to 0.757. Platt scaling improved probability reliability for several labels without systematically increasing AUC.
Conclusion:A CheXpert-trained DenseNet121 model showed moderate transferability to Mongolian chest radiographs, with strong performance for several clinically important findings and lower performance for other categories. Threshold optimization and calibration are feasible first-stage adaptation strategies. Larger multi-site Mongolian datasets, consensus labels, independent calibration/test sets, and prospective validation are required before clinical deployment.
- Full text:202611022152014224218. Монгол хүний цээжний рентген зурагт DenseNet121 загварын шийдвэрийн босго болон.pdf
