From setting to vetting: using artificial intelligence for Single Best Answer questions review
- Author:
Olivia NG
1
;
Siew Ping HAN
;
Magdalene Hui Min LEE
;
Dong Haur PHUA
Author Information
- Publication Type:Short Communication
- From:Korean Journal of Medical Education 2026;38(1):10-14
- CountryRepublic of Korea
- Language:English
-
Abstract:
Purpose:Maintaining the quality of Single Best Answer (SBA) questions remains a challenge in medical education, especially as artificial intelligence (AI)-generated items become more common. While considerable attention has been paid to AI question generation, the vetting process is under-explored and difficult to scale.
Methods:This study investigates the feasibility and reliability of using a large language model to support the vetting of SBA questions. An AI-based reviewer, QA-bot, was developed using custom GPT and embedded with 25 criteria aligned with Bloom’s taxonomy (Levels 1–3). QA-bot and two experienced educators independently evaluated 32 AI-generated SBA questions using the shared evaluation rubric.
Results:The rubric showed high internal consistency (Cronbach’s alpha=0.878), and strong inter-rater reliability between human reviewers (intraclass correlation coefficient [ICC]=0.893). QA-bot demonstrated good alignment with human raters (ICC=0.861 and 0.840). While the AI performed well on objective, rule-based criteria, it was less consistent in detecting irrelevant complexity and accurately judging difficulty.
Conclusion:These findings suggest that AI can function as an efficient first-pass reviewer, improving consistency and reducing workload, with human oversight remaining essential for educational and clinical relevance.
