Background: Summative assessment of student performance should provide information on achievement of program outcomes to support evaluation decisions. Alternative approaches to the traditional assessment systems like the written licensure examinations in Physical Therapy (PT) should be explored to ensure valid measurement of achievement of these terminal outcomes. Objective: The study aimed at establishing equivalence of two summative assessments new to PT in measuring achievement of the PT outcomes: work-based assessment using Entrustable Professional Activities (EPA) and knowledge-based assessment using Context-Dependent Item Sets (CDIS). Methodology: Thirty-two newly graduated PT's underwent a one-week EPA assessment and took a 102-item CDIS test (based on 14 clinical vignettes). Qualitative data from blueprint review, group face-to-face interviews with participants and assessors, and field notes from observations, and quantitative data from EPA entrustment decisions and CDIS scores were utilized to ascertain their comparability in terms of Purpose, Administration, Quality and Decisions. This was used to determine the extent of equivalence of the two assessments. Results: Review of both blueprints show alignment with PT outcomes, with integrative content motivating participants towards professional development. Administration were equally acceptable to users, though EPA had more practice opportunities with a longer assessment time. Entrustment decisions in EPA had a high inter-rater reliability, while CDIS had low reliability, with most items having poor discriminative power. Decisions of “pass” or “fail” had good concordance when high prevalence indices were considered. Conclusion There is high extent of equivalence in purpose of EPA and CDIS but are not equivalent in terms of administration. There is moderate equivalence in quality and decisions, with potential for increased concordance if improved quality of CDIS is attained.
Humans

BACKGROUND

Ahigh-quality measurement tool is essential to accurately assess the innovative teaching strategies in health professions education. The Blended Teaching Assessment Tool (BTAT) aims to evaluate quality blended teaching or instructional delivery in Philippine health science programs. However, there is a lack of studies examining students' cognitive processes to support the validity of questionnaires.

METHODOLOGY

Cognitive interviewing (CI) was employed to determine whether students interpreted and responded to the items correctly. Content analysis was done using Tourangeau's Cognitive framework. Four CIs were conducted by an expert moderator and note-taker with a total of 8 health science students (2 groups with 3 members, and 2 one-on-one interview) for around 1 to 2 hours via Zoom following a retroactive approach with verbal and spontaneous probing, guided by a semi-structured interview questionnaire.

RESULTS

Various issues related to comprehension, retrieval, judgment, and response were identified, leading to significant revisions of the tool from 82 items across 8 dimensions to 53 items across 5 dimensions. The challenges included unfamiliar terminology, ambiguous phrasing, complex statements, inconsistencies and irrelevance to students' real-life experiences. These findings emphasize the importance of students' feedback in enhancing the validity and reliability of assessment tools.

CONCLUSION

The Cognitive Interview identified crucial issues in comprehension, retrieval, judgment, and response, making it essential for developing the Blended Teaching Assessment Tool and ensuring valid responses on the quality of blended teaching and learning delivery.

Health Occupations ; Education ; Teaching