A melanoma diagnosis method based on large-scale vision-language models
10.16098/j.issn.0529-1356.2025.01.003
- VernacularTitle:基于大规模视觉语言模型的黑色素瘤诊断方法
- Author:
Jia-Yue ZHAO
1
;
Shi-Man LI
;
Chen-Xi ZHANG
Author Information
1. 复旦大学基础医学院数字医学研究中心,上海 200032;上海市医学图像处理与计算机辅助手术重点实验室,上海 200032
- Keywords:
Melanoma;
Large-scale vision-language model;
Fine-tuning;
Diagnosis;
Deep learning;
Human
- From:
Acta Anatomica Sinica
2025;56(1):22-29
- CountryChina
- Language:Chinese
-
Abstract:
Objective To develop a melanoma diagnosis framework based on large-scale vision-language models,and to explore the feasibility and accuracy of the framework for melanoma diagnosis.Methods The publicly available Derm7pt dataset,which was divided into a training set(346 cases),a validation set(161 cases),and a test set(320 cases)was utilized.A melanoma diagnosis framework based on large-scale vision-language models was proposed,comprising two text branches and one visual branch.In the text branches,one branch processed fixed clinical prompts,while the other handled learnable prompts.This design aimed to optimize the effectiveness of learnable prompts through guidance from fixed clinical prompts.The visual branch processed dermoscopic images and enhanced melanoma feature recognition through fine-tuning the image encoder.Results On the Derm7pt dataset,our method outperformd other existing method.It achieved an area under the receiver operating characteristic curve(AUC)of 87.35％,an accuracy of 84.17％,and an F1-score of 84.01％.Conclusion The study demonstrates that with appropriate fine-tuning strategies,methods based on large-scale vision-language pre-trained models can effectively adapt to melanoma diagnosis tasks.This approach can serve as a powerful auxiliary tool for doctors,helping them make more accurate diagnostic decisions.
