A Criterion-Referenced Examination of Physician Competence

Abstract

Despite the growing popularity of performance tests, scores on such measures have rarely been interpreted from a criterionreferenced perspective. This paper describes a test of skill in reading electrocardiographs (ECGs). Using generalizability theory, the errors of measurement and standard setting were estimated both alone and together from a criterionreferenced perspective. Performance on this test was also compared with a measure of the quality of training plus the multiplechoice questions and patient-management problems used in a medical certifying examination. Generalizabilityanalysesproduced positive results for the standard setting procedure and the ECGs, both separately and together. The preliminary validity evidence for scores was encouraging. The criterion-referenced ECGs ranked groups of examinees as expected based on prior education and examination experience. The criterion-referenced ECGs also had modest correlations with traditional measures of physician competence.