Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study
- Author:
Ariel Shana FREY-VOGEL
1
;
Kristina DZARA
;
Kimberly Anne GIFFORD
;
Yoon Soo PARK
;
Justin BERK
;
Allison HEINLY
;
Darcy WOLCOTT
;
Daniel Adam HALL
;
Shannon ELLIOTT SCOTT-VERNAGLIA
;
Katherine Anne SPARGER
;
Erica Ye-pyng CHUNG
Author Information
- Publication Type:Research article
- From:Journal of Educational Evaluation for Health Professions 2024;21(1):3-
- CountryRepublic of Korea
- Language:English
-
Abstract:
Purpose:Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence on the use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into 6 elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure.
Methods:Messick’s unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from 3 pediatric residency programs to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites was video-recorded during 2018–2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables).
Results:Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was 0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone (r=0.34, P=0.019).
Conclusion:Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.
