Further psychometric evaluations of a class-ranking model as a predictor of graduatesʼ clinical competence in the first year of residency

Abstract

To investigate further the psychometrics of a class-ranking model in which a weight of one-third was assigned to performance measures in basic sciences and a weight of two-thirds to ratings on six core clerkships. The first part of the study involved 215 graduates of Jefferson Medical College who--based on the ranking model--had been in the top and bottom quarters of the classes of 1991 and 1992. Six faculty, who did not know the graduates' ranks but were familiar with their performances and characteristics, were asked to judge the graduates' potential to become competent physicians. The graduates' ranks according to the model were then compared with the ratings they received from the faculty. The second part of the study investigated whether there was a linear relationship between class ranks and ratings of postgraduate competence, by using directors' ratings of the data-gathering skills of 598 graduates (1986-1990) at the end of their first year of residency. A concordance rate of 85% was obtained between the graduates' ranks and the ratings they received from the medical school faculty, which supports the criterion-related validity of the ranking model. In addition, class ranks were linearly related to ratings of postgraduate competence. However, women and graduates who had been low achievers in medical school were less likely to have given permission for collecting postgraduate ratings, which led to range restriction and a possible underestimation of the validity of the model. The psychometric evidence supports the class-ranking model, but other schools should exercise caution in employing the model until they accumulate evidence from data obtained from their own students.