Standard setting made easy: validating the Equal Z-score (EZ) method for setting cut-score for clinical examinations
Open Access
- 25 May 2020
- journal article
- research article
- Published by Springer Science and Business Media LLC in BMC Medical Education
- Vol. 20 (1), 1-9
- https://doi.org/10.1186/s12909-020-02080-x
Abstract
This study aims to assess the feasibility, reliability and validity of the panel-based Equal Z-score (EZ) method applied to objective structural clinical examination (OSCE) of Chinese medical students and undertaking a comparison with the statistical techniques-based Borderline Regression Method (BRM). Data received from two cohorts of 6th and 7th year medical students in Taiwan who set the mock OSCE as a formative assessment. Traditionally this medical school uses BRM to set the pass/fail cut-score. For the current study, 31 OSCE panellists volunteered to participate in the EZ method in parallel to the BRM. In the conduct of this study, each panel completed this task for an OSCE exam comprising 12 stations within less than 60 min. Moreover, none of the 31 panellists, whose are busy clinicians, had indicated that the task was too difficult or too time-consuming. Although EZ method yielded higher cut-scores than the BRM it was found reliable. Intraclass correlation (ICC) measuring absolute agreement, across the three groups of panellists was .893 and .937 for the first and second rounds respectively, demonstrating high level of agreement across groups with the EZ method and the alignment between the BRM and the EZ method was visually observed. The paired t-test results identified smaller differences between the cut-scores within methods than across methods. Overall this study suggests that the EZ method is a feasible, reliable and valid standard setting method. The EZ method requires relatively little resources (takes about an hour to assess a 12 station OSCE); the calculation of the cut-score is simple and requires basic statistical skills; it is highly reliable even when only 10 panellists participate in the process; and its validity is supported by comparison to BRM. This study suggests that the EZ method is a feasible, reliable and valid standard setting method.Keywords
Funding Information
- Ministry of Health and Welfare and Taiwan Association of Medical Education (MOST 106-2511-5010-001-MY3, 107F-M01-0603, 107QC018-2, V107EA-008, and V108EA-006)
This publication has 24 references indexed in Scilit:
- Reliability and Validity of Bookmark-Based Methods for Standard Setting: Comparisons to Angoff-Based Methods in the National Assessment of Educational ProgressEducational Measurement: Issues and Practice, 2011
- Comparing Construct Definition in the Angoff and Objective Standard Setting ModelsEducational and Psychological Measurement, 2011
- A standard setting method with the best performing students as point of reference: Practical and affordableMedical Teacher, 2010
- Is an Angoff Standard an Indication of Minimal Competence of Examinees or of Judges?Advances in Health Sciences Education, 2006
- Standard Setting in a Small Scale OSCE: A Comparison of the Modified Borderline-Group Method and the Borderline Regression MethodAdvances in Health Sciences Education, 2006
- The Bookmark Standard‐Setting Method: A Literature ReviewEducational Measurement: Issues and Practice, 2006
- Item Response Theory and Classical Test Theory: An Empirical Comparison of their Item/Person StatisticsEducational and Psychological Measurement, 1998
- A Comparison of the Nedelsky and Modified Angoff Standard-Setting Procedure On Evaluation OutcomeEducational and Psychological Measurement, 1981
- Judgment under Uncertainty: Heuristics and BiasesScience, 1974
- AN ITEM ANALYSIS WHICH TAKES INDIVIDUAL DIFFERENCES INTO ACCOUNTBritish Journal of Mathematical and Statistical Psychology, 1966