Increased accessibility of computer-based testing for residency application to a hospital in Brazil with item characteristics comparable to paper-based testing: a psychometric study
- Author:
Marcos Carvalho BORGES
1
;
Luciane Loures SANTOS
;
Paulo Henrique MANSO
;
Elaine Christine Dantas MOISÉS
;
Pedro Soler COLTRO
;
Priscilla Costa FONSECA
;
Paulo Roberto Alves GENTIL
;
Rodrigo de Carvalho SANTANA
;
Lucas Faria RODRIGUES
;
Benedito Carlos MACIEL
;
Hilton Marcos Alves RICZ
Author Information
- Publication Type:Technical report
- From:Journal of Educational Evaluation for Health Professions 2024;21(1):32-
- CountryRepublic of Korea
- Language:English
-
Abstract:
Purpose:With the coronavirus disease 2019 pandemic, online high-stakes exams have become a viable alternative. This study evaluated the feasibility of computer-based testing (CBT) for medical residency applications in Brazil and its impacts on item quality and applicants’ access compared to paper-based testing.
Methods:In 2020, an online CBT was conducted in a Ribeirao Preto Clinical Hospital in Brazil. In total, 120 multiple-choice question items were constructed. Two years later, the exam was performed as paper-based testing. Item construction processes were similar for both exams. Difficulty and discrimination indexes, point-biserial coefficient, difficulty, discrimination, guessing parameters, and Cronbach’s α coefficient were measured based on the item response and classical test theories. Internet stability for applicants was monitored.
Results:In 2020, 4,846 individuals (57.1% female, mean age of 26.64±3.37 years) applied to the residency program, versus 2,196 individuals (55.2% female, mean age of 26.47±3.20 years) in 2022. For CBT, there was an increase of 2,650 applicants (120.7%), albeit with significant differences in demographic characteristics. There was a significant increase in applicants from more distant and lower-income Brazilian regions, such as the North (5.6% vs. 2.7%) and Northeast (16.9% vs. 9.0%). No significant differences were found in difficulty and discrimination indexes, point-biserial coefficients, and Cronbach’s α coefficients between the 2 exams.
Conclusion:Online CBT with multiple-choice questions was a viable format for a residency application exam, improving accessibility without compromising exam integrity and quality.
