A Multimodal Deep Learning Model Integrating Clinical and Ultrasound Imaging Data for grading steatotic liver disease
10.11714/jsysu.med.YX20260026
- VernacularTitle:融合临床与超声图像多模态数据的深度学习模型评估脂肪肝分级
- Author:
Wenyi MENG
1
;
Xinrui YAO
2
;
Renjie LI
1
;
Jianqing PENG
3
;
Zuofeng XU
1
Author Information
1. Department of Ultrasound， the Seventh Affiliated Hospital， Sun Yat-sen University， Shenzhen， 518107， China
2. School of Electronic and Communication Engineering， Shenzhen Campus of Sun Yat-sen University， Shenzhen 518107， China
3. School of Intelligent Systems Engineering， Shenzhen Campus of Sun Yat-sen University， Shenzhen 518107， China
- Publication Type:Journal Article
- Keywords:
steatotic liver disease;
ultrasound images;
clinical data;
deep learning;
attenuation imaging
- From:
Journal of Sun Yat-sen University(Medical Sciences)
2026;47(3):528-538
- CountryChina
- Language:Chinese
-
Abstract:
ObjectiveThis study aimed to construct a neural network model integrating readily available clinical data and two-dimensional （2D） conventional ultrasound images to evaluate the severity of steatotic liver disease （SLD）. MethodsClinical data and ultrasound images were retrospectively collected from 649 patients who underwent ultrasound attenuation imaging （ATI）. Using ATI as the reference standard， patients were divided into four groups： normal （S0）， mild （S1）， moderate （S2）， and severe （S3） SLD. Data were randomly divided into training and validation sets at an 8∶2 ratio. A contrastive language-image pre-training （CLIP） model was utilized to extract features from both clinical and imaging data， which were then used to train random forest （RF） and multilayer perceptron （MLP） models. The diagnostic performance of these multimodal models was compared their single-modal counterparts in the validation set. ResultsThe MLP model outperformed the RF model （AUC［95%CI］： S0 = 0.96 ［0.93-0.99］， S1 = 0.99 ［0.96-1.00］， S2 = 0.75 ［0.64-0.82］， S3 = 0.88 ［0.81-0.93］）. Furthermore， the multimodal fusion model demonstrated significantly superior performance compared to single-modal models （clinical data only： AUC［95%CI］ of 0.89 ［0.81-0.95］， 0.69 ［0.52-0.79］， 0.64 ［0.43-0.67］ and 0.80 ［0.73-0.90］ for S0-S3， respectively； image data only： AUC［95%CI］ of 0.91 ［0.86-0.96］， 0.89 ［0.67-0.89］， 0.69 ［0.58-0.78］ and 0.86 ［0.82-0.94］）. Metrics including precision， recall， F1-score， F2-score， confusion matrices， and loss function learning curves further confirmed that multimodal data fusion significantly improved the predictive capability. ConclusionThe proposed CLIP-based MLP multimodal model can effectively and automatically grade SLD severity. This demonstrates the significant advantage of integrating clinical and ultrasound data， providing an accurate and reliable adjunctive tool for the clinical management of chronic SLD.
