Evaluating the quality and empathy of responses to patient questions on the Korean Academy of Periodontology’s online question and answer section:a cross-sectional study comparing periodontists and an AI-powered chatbot
10.5051/jpis.2402220111
- Author:
Jae-Hong LEE
;
So-Hae OH
;
Falk SCHWENDICKE
;
Akhilanand CHAURASIA
;
Young-Taek KIM
- Publication Type:Research Article
- From:Journal of Periodontal & Implant Science
2025;55(6):485-496
- CountryRepublic of Korea
- Language:English
-
Abstract:
Purpose:This study aimed to evaluate and compare the responses of an artificial intelligence (AI)–powered chatbot and professional periodontists to patient queries in periodontology and implantology, using the Korean Academy of Periodontology’s (KAP) online question and answer (Q&A) section.
Methods:In this comparative cross-sectional study, we analyzed 219 patient-submitted periodontal and implant knowledge questions from the KAP online Q&A section. A panel of 10 evaluators—5 periodontists and 5 laypersons—rated both the periodontist’s and the AI chatbot’s responses using standardized scales. We applied the t-test and Spearman correlation coefficients to compare response quality, empathy, consistency, and evaluator preferences.
Results:Ten evaluators judged the AI chatbot’s responses to be significantly superior in quality and empathy compared to periodontist replies. A higher proportion of periodontist responses fell below acceptable quality (“very poor” or “poor”) than chatbot responses (28.7% vs. 15.0%; P<0.001), and more chatbot replies were rated “empathetic” or “very empathetic” (62.5% vs. 42.8%; P<0.001). Overall response consistency was deemed satisfactory at 64.2%, with no significant difference in consistency or preference between periodontist and lay evaluators.
Conclusions:AI-powered chatbots can deliver more accurate and empathetic answers than human periodontists, suggesting their potential role as consultation assistants merits further investigation. The high intraclass correlation coefficient values (0.79–0.93) indicate a high level of agreement among evaluators in both the periodontist and lay evaluator groups, thus confirming the reliability and robustness of the study’s assessment methodology.
