PURPOSE: Test equating studies in medical education have been conducted only for high-stake exams or to compare two tests given in a single course. Based on item response theory, we equated computer-based test (CBT) results from the basic medical education curriculum at the College of Medicine, the Catholic University of Korea and evaluated the validity of using fixed passing scores. METHODS: We collected 232 CBTs (28,636 items) for 40 courses administered over a study period of 9 years. The final data used for test equating included 12 pairs of tests. After test equating, Wilcoxon rank-sum tests were utilized to identify changes in item difficulty between previous tests and subsequent tests. Then, we identified gaps between equated passing scores and actual passing scores in subsequent tests through an observed-score equating method. RESULTS: The results of Wilcoxon rank-sum tests indicated that there were no significant differences in item difficulty distribution by year for seven pairs. In the other five pairs, however, the items were significantly more difficult in subsequent years than in previous years. Concerning the gaps between equated passing scores and actual passing scores, equated passing scores in 10 pairs were found to be lower than actual passing scores. In the other two pairs, equated passing scores were higher than actual passing scores. CONCLUSION: Our results suggest that the item difficulty distributions of tests taught in the same course during successive terms can differ significantly. It may therefore be problematic to use fixed passing scores without considering this possibility.
Curriculum ; Education, Medical ; Educational Measurement ; Korea ; Methods

BACKGROUND: The clinical practice examination (CPX) was introduced in 2010, and the Seoul-Gyeonggi CPX Consortium developed the patient-physician interaction (PPI) assessment tool in 2004. Both institutions use rating scales on classified sections of PPI but differ in their scoring of key components. This study investigated the accuracy of standardized patient scores across rating scales by comparing checklist methods and verified the concurrent validity of two comparable PPI rating tools. METHODS: An educational CPX module dyspepsia case was administered to 116 fourth-year medical students at Hanyang University College of Medicine. One experienced standardized patient rated exams using two different PPI scales. She scored checklists composed of 43 items related to the two original PPI scales through video clips of the same students. From these checklists, we calculated Pearson's correlation coefficient. RESULTS: The correlations of total PPI score between the checklist and rating scale methods were 0.29 for the Korean Medical Licensing Examination (KMLE) tool and 0.30 for the consortium tool. The correlations between the KMLE and consortium tools were 0.74 for checklists and 0.83 for rating scales. In terms of section scores, the consortium tool showed only three significant correlations between the two methods out of seven sections and the KMLE tool showed only two statistically significant correlations out of five sections. CONCLUSION: The rating scale and checklist methods exhibited a weak relationship in the PPI assessment, but a high correlation between assessment tools using the same method. However, the current rating scale requires modification by reorganizing key scoring components through factor analysis.
Behavior Rating Scale ; Checklist* ; Dyspepsia ; Education, Medical ; Educational Measurement ; Humans ; Licensure ; Methods ; Physician-Patient Relations ; Students, Medical ; Weights and Measures

PURPOSE: This study aims to develop a system of course monitoring by students and evaluate the course quality management system (CQMS) implemented as an educational assessment tool. METHODS: This research was conducted in accordance with the ADDIE model which is a well-known instructional design model. The ADDIE process includes needs analysis, design of the course monitoring and course evaluation, development of evaluation forms, implementation of course monitoring, and evaluation of the program. RESULTS: To meet the need for a system that can replace the traditional lecture evaluation approach, this study developed and implemented a new course evaluation system. In comparison with the quantitative evaluation method, course monitoring by students provided more qualitative information on classes and courses from the students' perspective. The students' realistic description helped know how student felt the atmosphere of class and what kinds of teaching style students preferred. However, some view that the evaluation by the four monitoring members was less reliable. CONCLUSION: This study proposed a course quality management based on students' monitoring which emphasized the narrative evaluation to help to identify the strengths and weaknesses of the classes and gather qualitative information from the students' perspective that can be used to improve the courses. It is expected that providing the monitoring members with better orientation could help manage the quality of the courses using the monitoring system.
Atmosphere ; Educational Measurement ; Evaluation Studies as Topic ; Humans ; Methods ; Schools, Medical*

PURPOSE: This study aimed to assess the agreement between 2 raters in evaluations of students on a prosthodontic clinical practical exam integrated with directly observed procedural skills (DOPS). METHODS: A sample of 76 students was monitored by 2 raters to evaluate the process and the final registered maxillomandibular relation for a completely edentulous patient at Mansoura Dental School, Egypt on a practical exam of bachelor’s students from May 15 to June 28, 2017. Each registered relation was evaluated from a total of 60 marks subdivided into 3 score categories: occlusal plane orientation (OPO), vertical dimension registration (VDR), and centric relation registration (CRR). The marks for each category included an assessment of DOPS. The marks of OPO and VDR for both raters were compared using the graph method to measure reliability through Bland and Altman analysis. The reliability of the CRR marks was evaluated by the Krippendorff alpha ratio. RESULTS: The results revealed highly similar marks between raters for OPO (mean= 18.1 for both raters), with close limits of agreement (0.73 and −0.78). For VDR, the mean marks were close (mean= 17.4 and 17.1 for examiners 1 and 2, respectively), with close limits of agreement (2.7 and −2.2). There was a strong correlation (Krippendorff alpha ratio, 0.92; 95% confidence interval, 0.79–0.99) between the raters in the evaluation of CRR. CONCLUSION: The 2 raters' evaluation of a clinical traditional practical exam integrated with DOPS showed no significant differences in the evaluations of candidates at the end of a clinical prosthodontic course. The limits of agreement between raters could be optimized by excluding subjective evaluation parameters and complicated cases from the examination procedure.
Centric Relation ; Dental Occlusion ; Education, Dental ; Educational Measurement ; Egypt ; Humans ; Methods ; Prosthodontics ; Schools, Dental ; Vertical Dimension

PURPOSE: Peer assessment provides a framework for developing expected skills and receiving feedback appropriate to the learner's level. Near-peer (NP) assessment may elevate expectations and motivate learning. Feedback from peers and NPs may be a sustainable way to enhance student assessment feedback. This study analysed relationships among self, peer, NP, and faculty marking of an assessment and students' attitudes towards marking by those various groups. METHODS: A cross-sectional study design was used. Year 2 osteopathy students (n=86) were invited to perform self and peer assessments of a clinical history-taking and communication skills assessment. NPs and faculty also marked the assessment. Year 2 students also completed a questionnaire on their attitudes to peer/NP marking. Descriptive statistics and the Spearman rho coefficient were used to evaluate relationships across marker groups. RESULTS: Year 2 students (n=9), NPs (n=3), and faculty (n=5) were recruited. Correlations between self and peer (r=0.38) and self and faculty (r=0.43) marks were moderate. A weak correlation was observed between self and NP marks (r=0.25). Perceptions of peer and NP marking varied, with over half of the cohort suggesting that peer or NP assessments should not contribute to their grade. CONCLUSION: Framing peer and NP assessment as another feedback source may offer a sustainable method for enhancing feedback without overloading faculty resources. Multiple sources of feedback may assist in developing assessment literacy and calibrating students' self-assessment capability. The small number of students recruited suggests some acceptability of peer and NP assessment; however, further work is required to increase its acceptability.
Australia ; Cohort Studies ; Cross-Sectional Studies ; Educational Measurement ; Humans ; Learning ; Literacy ; Methods ; Osteopathic Medicine ; Peer Review ; Self-Assessment

PURPOSE: The purpose of this retrospective study was to evaluate students' academic outcomes after implementation of the team-based learning (TBL) approach in patient/client management courses in an entry-level doctor of physical therapy (DPT) curriculum. METHODS: The research design of this study involved comparing written and practical exam scores from DPT student cohorts taught with the traditional instructional methods (lecture-based) to those of students from subsequent cohorts taught using the TBL approach in two patient/client management courses: basic skills and cardiopulmonary. For this comparison, the exams used, the number of contact hours and labs, and the instructors who taught these courses remained the same during the transition between these two instructional methods (traditional vs. TBL). The average of all individual course exam scores was used for data analysis. RESULTS: In both courses, there were no meaningful differences in the mean exam scores among students across years of cohorts receiving the same instructional method, which allowed clustering students from different years of cohorts in each course receiving the same instructional method into one group. For both courses, the mean exam score was significantly higher in the TBL group than in the traditional instruction group: basic skills course (P<0.001) and cardiopulmonary course (P<0.001). CONCLUSION: Student cohorts taught using the TBL approach academically outperformed those who received the traditional instructional method in both entry–level DPT patient/client management courses.
Cohort Studies ; Curriculum* ; Education, Professional ; Educational Measurement ; Health Occupations ; Humans ; Learning* ; Methods ; Physical Therapists ; Research Design ; Retrospective Studies ; Statistics as Topic ; United States*

OBJECTIVES: The purpose of this study was to examine the effect of an educational program using priming on oral health, and dental plaque, and gingival bleeding indices in an elderly population with dementia (EWD) residing in nursing homes. METHODS: A pretest and posttest were conducted with a nonequivalent control group. In total, 40 participants from an EWD had the following characteristics: 1) they were aged >65 years, 2) they scored <23 points on the MMSE-DS, 3) they were able to communicate, 4) they were able to hold a toothbrush and brush his or her teeth by themselves, and 5) they had no periodontal treatment for the last 3 months. Through random sampling, participants were assigned into either the experimental (n=20) or control group (n=20). The educational program consisted of twelve 30-minute sessions (twice per week for 6 weeks). The control group was demonstrated the standard tooth-brushing method by the facility during the study period. RESULTS: There were no significant differences in demographics or main variables at pretest. The experimental group showed significant differences in the subjective oral health condition, as compared to the control group. The experimental group showed an increase in the level of tooth-brushing effort (t=―8.816, P<.001) and higher scores of self-evaluative dental health status (P<.001) as compared to the control group. The experimental group showed a reduced plaque index (t=11.179, P<.001) and gingival bleeding index (t=5.812, P<.001) compared to the control group. Additionally, the experimental group showed significant differences in changes of plaque index and gingival bleeding index between the pretest and posttest. CONCLUSIONS: An interventional program on tooth brushing with priming effectively improved dental health, plaque index, and gingival bleeding index in EWDs residing in nursing homes. Direct care providers should understand the remaining abilities of EWDs and carry out priming when providing care to them.
Aged* ; Dementia* ; Demography ; Dental Plaque ; Educational Measurement ; Hemorrhage ; Humans ; Memory ; Methods ; Nursing Homes* ; Nursing* ; Oral Health ; Repetition Priming ; Tooth

INTRODUCTIONMetacognition is a cognitive debiasing strategy that clinicians can use to deliberately detach themselves from the immediate context of a clinical decision, which allows them to reflect upon the thinking process. However, cognitive debiasing strategies are often most needed when the clinician cannot afford the time to use them. A mnemonic checklist known as TWED (T = threat, W = what else, E = evidence and D = dispositional factors) was recently created to facilitate metacognition. This study explores the hypothesis that the TWED checklist improves the ability of medical students to make better clinical decisions.

METHODSTwo groups of final-year medical students from Universiti Sains Malaysia, Malaysia, were recruited to participate in this quasi-experimental study. The intervention group (n = 21) received educational intervention that introduced the TWED checklist, while the control group (n = 19) received a tutorial on basic electrocardiography. Post-intervention, both groups received a similar assessment on clinical decision-making based on five case scenarios.

RESULTSThe mean score of the intervention group was significantly higher than that of the control group (18.50 ± 4.45 marks vs. 12.50 ± 2.84 marks, p < 0.001). In three of the five case scenarios, students in the intervention group obtained higher scores than those in the control group.

CONCLUSIONThe results of this study support the use of the TWED checklist to facilitate metacognition in clinical decision-making.

Adult ; Checklist ; Clinical Decision-Making ; methods ; Education, Medical ; methods ; Educational Measurement ; Female ; Humans ; Malaysia ; Male ; Metacognition ; Prejudice ; Program Evaluation ; Students, Medical ; Universities ; Young Adult

INTRODUCTIONThis study examines the relationship between the clinical performance of medical students and their performance as doctors during their internships.

METHODSThis retrospective study involved 63 applicants to a residency programme conducted at the Chonnam National University Hospital, South Korea, in November 2012. We compared the performance of the applicants during their internship with the clinical performance of the applicants during their fourth year of medical school. The performance of the applicants as interns was periodically evaluated by the faculty of each department, while the clinical performance of the applicants as fourth year medical students was assessed using the Clinical Performance Examination (CPX) and the Objective Structured Clinical Examination (OSCE).

RESULTSThe performance of the applicants as interns was positively correlated with their clinical performance as fourth year medical students, as measured by CPX and OSCE. The performance of the applicants as interns was moderately correlated with the patient-physician interactions items addressing communication and interpersonal skills in the CPX.

CONCLUSIONThe clinical performance of medical students during their fourth year in medical school was related to their performance as medical interns. Medical students should be trained to develop good clinical skills, through actual encounters with patients or simulated encounters using manikins, so that they are able to become competent doctors.

Clinical Competence ; Education, Medical, Graduate ; methods ; Educational Measurement ; methods ; Female ; Humans ; Internship and Residency ; Male ; Republic of Korea ; Retrospective Studies ; Schools, Medical ; Students, Medical ; psychology

PURPOSE: Academic debate is an effective method to enhance the competences of critical thinking, problem solving, communication skills and cooperation skills. The present study examined the improvement of debate competence which is an outcome of debate-based flipped learning. METHODS: A questionnaire was administrated to second-year premedical school students at Yeungnam University. In total 45 students participated in the survey. The survey questionnaire was composed of 60 items of eight subfactors on debate competence. To investigate the homogeneous of low and high achievement groups, 18 items on empathy and 75 items on critical thinking scales were used. To compare the pretest with posttest scores, data was analyzed using paired sample t-test. RESULTS: There were no significant differences between low and high achievement groups by average grade at the beginning of the semester. There was a significant improvement in high achievers on the logical argumentation (p<0.001), proficiency in inquiry (p<0.01), active participation (p<0.001), ability to investigate and analyze (p<0.001), observance of debate rules (p<0.05), and acceptability (p<0.05). Even in low achievers, active participation (p<0.05) and ability to investigate and analyze (p<0.01) were significantly improved. CONCLUSION: Results showed that students could improve their debate competence by the debate-based flipped learning. A prospective and comparative study on the communication and teamwork competences needs to be conducted in the future. It is suggested that in-depth discussion for the curriculum design and teaching will be needed in terms of the effectiveness and the outcomes of the medical humanities.
*Achievement ; *Clinical Competence ; Communication ; Cooperative Behavior ; *Curriculum ; Education, Premedical/*methods ; Educational Measurement ; Empathy ; Female ; *Humanities ; Humans ; Logic ; Male ; *Problem Solving ; *Problem-Based Learning ; Republic of Korea ; Schools, Medical ; Students, Premedical ; Surveys and Questionnaires ; Universities