A multicenter evaluation study of the use of large language models in neuro-ophthalmology
10.13389/j.cnki.rao.2025.0139
- VernacularTitle:大语言模型在神经眼科中应用的多中心评价
- Author:
Zixun WANG
1
;
Xiaoling ZHANG
;
Hongqiang JIA
;
Ruihua WEI
;
Yuhang WANG
;
Ke FAN
;
Yanhua QI
;
Xueshuo XIE
;
Shihui WEI
;
Zhiqing LI
Author Information
1. 300384 天津市,天津医科大学眼科医院、眼视光学院、眼科研究所,国家眼耳鼻喉疾病临床医学研究中心天津市分中心,天津市视网膜功能与疾病重点实验室
- Publication Type:Journal Article
- Keywords:
neuro-ophthalmology;
artificial intelligence;
large language model;
application evaluation;
multicenter
- From:
Recent Advances in Ophthalmology
2025;45(10):810-815
- CountryChina
- Language:Chinese
-
Abstract:
Objective To evaluate answers to typical clinical questions related to neuro-ophthalmology generated by Artificial Intelligence(AI)Large Language Models(LLM)and to explore the performance of neuro-ophthalmology-related questions on LLM in a multidimensional manner using objective and expert assessment.Methods Multicenter,random-ized,cross-sectional pilot study.Thirty typical questions related to neuro-ophthalmology were selected based on four per-spectives:definition,etiology,clinical manifestations and signs,and treatment and prognosis,and were analyzed quantita-tively using Deepseek,Wenxin Yiyin 4.0,Doubao,and Kimi 1.5,which are four open-source LLMs in China,and quantita-tively analyzed with objective assessment;and quantitatively rated by three ophthalmologists using expert assessment for 120 answer texts.Three ophthalmology experts quantitatively scored the 120 answer texts.Three ophthalmologists quantita-tively scored the 120 answer texts.Level 3,5,and 4 Likert scales were developed according to the completeness,accura-cy,professionalism,relevance,and criticality of the question texts,respectively.The best-performing LLM was selected,and its performance was observed across the four types of questions.Additionally,three other experts assessed whether the best-performing one could be evaluated as a substitute for real-world doctor-patient communication.Results In the objective Chinese text reading difficulty analysis,the differences in total word count among the four LLMs were statistically significant(all P＜0.001).Of the four LLMs,Kimi 1.5 performed the best,with frequencies of 61％,29％,and 41％for the highest scores in completeness(3),accuracy and professionalism(5),and relevance and usefulness(4),respective-ly.Kimi 1.5 performed more consistently on the questions on the four areas of neuro-ophthalmologic disorders:definition,etiology,clinical manifestations and signs,treatment,and prognosis,with no between-group differences(P＞0.05).Con-clusion Chinese language LLMs have great potential in the clinical application of neuro-ophthalmology.Kimi 1.5 outper-forms other LLMs in terms of completeness,accuracy,professionalism,relevance,and usefulness,but it still cannot re-place real-world doctor-patient communication.There is a need to explore new diagnostic and therapeutic model of AI+physician in the future.
