An investigation of digestive healthcare workers and new chatbots on knowledge of endoscopic screening of gastrointestinal cancers
10.3760/cma.j.cn321463-20230424-00087
- VernacularTitle:消化医护人员与新型聊天机器人在消化内镜癌症筛查相关知识的调查表现
- Author:
Yaling LIU
1
;
Yong LYU
;
Chenxi KANG
;
Xiangping WANG
;
Jing LI
;
Ling WANG
;
Haiying WANG
;
Yongzhan NIE
;
Kaichun WU
;
Yanglin PAN
Author Information
1. 空军军医大学西京消化病医院 国家消化系统疾病临床医学研究中心 消化系肿瘤整合防治全国重点实验室，西安 710032
- Keywords:
Endoscopy, digestive system;
Esophageal neoplasms;
Stomach neoplasms;
Colorectal neoplasms;
Screening;
Machine learning
- From:
Chinese Journal of Digestive Endoscopy
2023;40(11):892-899
- CountryChina
- Language:Chinese
-
Abstract:
Objective:To compare the knowledge of endoscopic screening of gastrointestinal cancers between digestive healthcare workers and new chatbots (chatGPT and new Bing).Methods:A test with twenty-three questions of endoscopic screening of gastrointestinal cancers was conducted, focusing on the appropriate age of screening, high-risk factors, the follow-up time, and the advantages and risks of digestive endoscopy. Digestive healthcare workers were invited to complete the test through electronic questionnaires. New Bing and chatGPT were used to answer each question for 10 rounds. The primary endpoint was the correct rate of all answers. The answer accuracy between digestive healthcare workers and new chatbots were compared using variance analysis, and the factors that affected the accuracy of the answers in digestive healthcare workers were explored using univariate and multivariable liner regression analysis.Results:The results of the test completed by 76 digestive healthcare workers (21 residents, 28 digestive nurses, and 27 digestive doctors) were analyzed. The accuracies were 36.4%±10.9%, 34.5%±10.2%, 52.2%± 12.6%, 46.3%±9.8% and 67.1%±9.3% in residents, digestive nurses, digestive doctors, chatGPT, and new Bing, respectively, with significant difference ( F=22.6, P<0.001). The accuracy was highest in new Bing ( P<0.001). The accuracy was comparable between chatGPT and digestive doctors (LSD- t=-1.398, P=0.166), and both higher than that of digestive nurses (LSD- t=2.956, P=0.004; LSD- t=5.955, P<0.001) and residents (LSD- t=2.402, P=0.018; LSD- t=4.951, P<0.001). Furthermore, the accuracy was comparable between digestive nurses and residents (LSD- t=-0.574, P=0.567). Compared with new Bing, digestive doctors had lower accuracy in answering questions related to adverse events of screening, follow-up recommendation of intestinal metaplasia, high risk factors and screening methods for colon cancer ( P<0.05), but higher accuracy in answering questions related to endoscopic adverse events and screening methods for esophageal cancer ( P<0.05). Multivariable liner regression analysis showed that being digestive doctors ( β=11.7, t=3.054, P=0.003) and questionnaire response time (≥7.6 min) ( β=7.8, t=2.894, P=0.005) were independent factors for the answer accuracy of digestive healthcare workers. Conclusion:Compared with digestive healthcare workers, New chatbots—new Bing has higher accuracy in answering gastrointestinal cancer screening-related questions, but performs poorly in answering questions such as adverse events of endoscopy and screening methods for esophageal cancer.
