Application of multi-reader multi-case design in evaluating artificial intelligence-assisted imaging diagnostic trials
10.16781/j.CN31-2187/R.20240775
- VernacularTitle:多阅片者多病例设计在人工智能辅助阅片影像诊断试验评价中的应用
- Author:
Huiqin WAN
1
;
Man XIANG
;
Zhemin PAN
;
Yingyi QIN
;
Qian HE
;
Jia HE
Author Information
1. 同济大学医学院,上海 200092
- Keywords:
artificial intelligence;
multi-reader multi-case design;
Obuchowski-Rockette method;
rib fractures;
diagnostic accuracy
- From:
Academic Journal of Naval Medical University
2025;46(4):504-510
- CountryChina
- Language:Chinese
-
Abstract:
Objective To evaluate the clinical efficacy of artificial intelligence(AI)-assisted imaging diagnostic trials using multi-reader multi-case(MRMC)design,so as to provide a scientific basis for clinical evaluation of imaging diagnostic trials.Methods The MRMC design,widely used in imaging diagnostic trials,was adopted in this study.The Obuchowski-Rockette(OR)method of MRMC design was detailed,including model construction and test methods.A case study was conducted,collecting imaging data of 200 subjects from 3 hospitals,with 133 cases of rib fractures and 68 cases of non-rib fractures.Three radiologists reviewed all CT images of the subjects.The area under curve(AUC)value,sensitivity and specificity in detecting rib fractures between 2 reading modalities(radiologists with AI assistance vs radiologists reading independently)were compared.Results The AI-assisted reading group had an AUC value of 0.958,while the radiologist-independent reading group had an AUC value of 0.902,showing a significant difference(P＜0.001).The overall sensitivity and specificity of the AI-assisted reading group were 0.970 and 0.946,respectively;while the sensitivity and specificity of the radiologist-independent reading group were 0.838 and 0.966,respectively.The difference of sensitivity between groups was 0.131(95％confidence interval[CI]0.091-0.171),and the difference of specificity was-0.020(95％CI-0.059-0.020),indicating a significant difference in sensitivity but not in specificity between AI-assisted and radiologist-independent reading groups.Both groups had positive likelihood ratios(+LR)greater than 10 and negative likelihood ratios(-LR)less than 0.2,with positive predictive values approaching 1,suggesting that the diagnostic accuracy of the AI-assisted imaging diagnostic trials was high.Conclusion The AI-assisted reading method demonstrates a significant advantage in enhancing diagnostic efficiency,not only improving the diagnostic accuracy and detection rate of rib fractures,but also improving the work efficiency of radiologists and optimizing hospital services.
