BACKGROUND

Ahigh-quality measurement tool is essential to accurately assess the innovative teaching strategies in health professions education. The Blended Teaching Assessment Tool (BTAT) aims to evaluate quality blended teaching or instructional delivery in Philippine health science programs. However, there is a lack of studies examining students' cognitive processes to support the validity of questionnaires.

METHODOLOGY

Cognitive interviewing (CI) was employed to determine whether students interpreted and responded to the items correctly. Content analysis was done using Tourangeau's Cognitive framework. Four CIs were conducted by an expert moderator and note-taker with a total of 8 health science students (2 groups with 3 members, and 2 one-on-one interview) for around 1 to 2 hours via Zoom following a retroactive approach with verbal and spontaneous probing, guided by a semi-structured interview questionnaire.

RESULTS

Various issues related to comprehension, retrieval, judgment, and response were identified, leading to significant revisions of the tool from 82 items across 8 dimensions to 53 items across 5 dimensions. The challenges included unfamiliar terminology, ambiguous phrasing, complex statements, inconsistencies and irrelevance to students' real-life experiences. These findings emphasize the importance of students' feedback in enhancing the validity and reliability of assessment tools.

CONCLUSION

The Cognitive Interview identified crucial issues in comprehension, retrieval, judgment, and response, making it essential for developing the Blended Teaching Assessment Tool and ensuring valid responses on the quality of blended teaching and learning delivery.

Health Occupations ; Education ; Teaching

BACKGROUND

Due to the COVID-19 outbreak, schools had to switch online. The sudden transition to blended teaching and learning (BTL) poses challenges for students and teachers, especially for health science programs that require hands-on practical experience. The validity, reliability, and responsiveness of these self-report questionnaires (SRQs) should be established to ensure the accuracy of the results as intended by the SRQ.

OBJECTIVES

This study critically appraised, compared, and summarized the psychometric properties of SRQ evaluating BTL among health science university students. This review determined the SRQ’s reliability, internal consistency, various forms of validity (content, criterion, construct), and responsiveness.

METHODS

Following a 10-step procedure based on COSMIN guidelines, we conducted a systematic review of SRQs used by health science university students to evaluate blended teaching and learning. Studies were eligible if they reported psychometric properties of SRQs related to blended learning among university health science students; exclusions included studies focusing on perceptions, attitudes, self-efficacy, and satisfaction, as well as articles such as biographies, editorials, and conference materials. Searches covered multiple electronic databases until April 26, 2023, including PubMed, EMBASE, Web of Science, MEDLINE (OVID), PsycInfo, CINAHL, EBSCOHOST, ERIC, Scopus, Science Direct, Google Scholar, JSTOR, Acta Medica Philippina, Philippine Journal of Health Research and Development, and HERDIN, managed through Zotero. Two independent reviewers performed database searches, title and abstract screening, and full-text evaluations, with a third reviewer resolving any disputes. The COSMIN Risk of Bias Checklist was employed to evaluate included studies on the development and various measurement properties of SRQs. The reviewers assessed SRQ standards, including validity, reliability, internal consistency, measurement error, responsiveness, interpretability, and feasibility. Data extraction and result tabulation were independently completed, with content comparison by two health education experts. This evaluation categorized the SRQs into three quality and validity levels.

RESULTS

The study examined five articles; four were rated as 'doubtful' and one as 'inadequate' in the overall development of SRQ. All four 'doubtful' studies demonstrated questionable content validity when university students were asked about the questionnaire's relevance, comprehensiveness, and comprehensibility. Only half of these studies achieved an 'adequate' rating for content validity based on expert opinions on relevance and comprehensiveness. All but one study scored from 'very good' to 'adequate' in structural validity. Three out of the four studies scored a very good rating for internal consistency, while one was deemed 'inadequate' in internal consistency, cross-cultural validity, and reliability. Three out of four studies scored 'very good' on construct validity, but all overlooked criterion validity and responsiveness. Conducted in various locations, including Australia, Romania, Turkey, and Taiwan, these studies highlighted both common characteristics and limitations in questionnaire development according to the COSMIN guidelines. Four studies were deemed reliable and valid for BTL constructs (Category A); Wu et al. requires further validation (Category B). Study limitations included heterogeneity in populations, settings, and questionnaire versions, potential subjective bias in SRQ content comparison, and the evolving nature of SRQs in blended learning contexts.

CONCLUSION

The systematic review reports the development and evaluation of SRQs for BTL while identifying gaps in their applicability to health science programs. The Blended Learning Scale (BLS) of Lazar et al. and the Blended Learning Questionnaire (BLQ) of Ballouk et al. showed an ‘adequate' rating for content validity. BLS revealed very good structural validity, internal consistency, and adequate content validation. Although the BLQ lacked Confirmatory Factor Analysis, it yielded valuable constructs for evaluating health sciences students' experiences in BTL. Both tools require improvements on recall period, completion time, interpretability, and feasibility. The review underscores the necessity for cont inuous assessment and enhancement of such instruments in BTL, advocating a rigorous scale development process. Furthermore, it encourages the customization of teaching and learning evaluation tools to suit specific institutional contexts while promoting further validation of these questionnaires across different populations in future research.

Human ; Psychometrics ; Checklist ; Self Report ; Universities ; Health Education