Comparative audit capabilities of five Chinese large language models in detecting drug-drug interaction in outpatient prescriptions
- VernacularTitle:5种中文大语言模型对门诊处方中药物相互作用的审核能力比较
- Author:
Zihao WANG
1
;
Jinbao LI
1
;
Yan ZHANG
2
;
Sen YAN
1
Author Information
1. Dept. of Pharmacy，Sanmenxia Hospital of the Yellow River，Henan Sanmenxia，472000，China
2. Dept. of Ophthalmology，Sanmenxia Hospital of the Yellow River，Henan Sanmenxia，472000，China
- Publication Type:Journal Article
- Keywords:
large language models;
drug-drug interaction;
Prescription review;
Sensitivity;
auxiliary prescription-review tools;
Outpatient prescription
- From:
China Pharmacy
2026;37(19):2577-2581
- CountryChina
- Language:Chinese
-
Abstract:
OBJECTIVE To provide a reference for the optimization of prescription-review software and the selection of auxiliary prescription-review tools for medical institutions.METHODS A total of 559 eligible prescriptions were collected from the Meikang Prescription Automatic Screening System of a tertiary grade-A hospital from January 2025 to March 2026. Judged according to the composite reference standard （CRS） established in this study， there were 206 positive prescriptions [with drug-drug interaction （DDI）] and 353 negative prescriptions （without DDI）. DeepSeek， Doubao， Wenxin yiyan， Kimi and Tongyi qianwen were separately adopted to identify DDI in the 559 prescriptions， and binary judgment was performed on their output texts related to DDI. Taking the results judged by CRS as the gold standard， the confusion matrix was constructed and model performance indicators were calculated. The differences in sensitivity between any two large language models were compared， the relative sensitivity （RS） and its 95% confidence interval （CI） were calculated， and the results were visualized by a forest plot.RESULTS Wenxin yiyan achieved the highest sensitivity （74.8%） with the lowest risk of missed reports， whereas its specificity was only 40.8%. It had the lowest accuracy （53.3%） and presented a high risk of false positive reports. Tongyi qianwen obtained the highest accuracy （65.3%） and the optimal F1-score （0.587）. DeepSeek showed moderate performance in both sensitivity and specificity. Doubao and Kimi exhibited relatively low sensitivity （38.8% and 45.1%， respectively）， accompanied by high risks of missed reports. Pairwise comparison revealed that the sensitivity of Wenxin yiyan was significantly higher than that of DeepSeek [ χ 2 ＝4.84， P ＝0.028； RS＝1.124， 95%CI （1.018， 1.230）]. The sensitivity of Tongyi qianwen was significantly higher than that of Doubao [ χ 2 ＝4.77， P ＝0.029； RS＝1.239， 95%CI （1.035， 1.443）]. DeepSeek had significantly higher sensitivity than Doubao [ χ 2 ＝37.65， P ＜0.001； RS＝1.715， 95%CI （1.491， 1.939）]. No statistically significant differences in sensitivity were observed between Kimi versus Doubao and between Tongyi qianwen versus Kimi （ P ＞0.05）.CONCLUSIONS Wenxin yiyan is applicable for preliminary-screening scenarios with low tolerance for missed reports and requires combined manual review. Tongyi qianwen is suitable for auxiliary outpatient prescription review. DeepSeek can act as a transitional solution for medical institutions without dedicated prescription-review software. Doubao and Kimi are not recommended for independent application in prescription DDI audit.
