The Reliability and Validity of the American Board of Internal Medicine Monthly Evaluation Form

Abstract

To report indicators of reliability and validity of the American Board of Internal Medicine Evaluation Form (ABIM-MEF) at one institution (Wright-Patterson Medical Center). Completed ABIM-MEFs from 1990-1999 were reviewed. Reliability measures included Cronbach α, interrater reliability, and rating consistency between different types of staff and rotations. Construct validity was investigated by tracking ABIM-MEF scores over time and with factor analysis. Predictive validity was assessed by correlating ABIM-MEF scores with the In-training Examination and ABIM Certifying Examination results. The 71 residents averaged 12 ABIM-MEFs per year. The forms had a Cronbach α of 0.96 and high interrater reliability (intraclass correlation coefficients > 0.80). Ratings did not differ by type of attending or rotation, except that noninternists rated residents lower on procedural skills. ABIM-MEF questions about judgment, knowledge, and clinical skills showed significant improvement from month to month during each academic year as well as year to year. In contrast, questions on professional attitudes, humanism, and procedural skills sections improved between postgraduate year 1 and postgraduate year 2 only. ABIM-MEF questions collapsed into two domains in factor analysis: judgment-knowledge-skills and attitude-humanism. ABIM-MEF questions from judgment and knowledge sections modestly predicted In-training Examination and ABIM Certifying Examination results. In contrast, professional attitude, humanism, and clinical as well as procedural skill questions had little discriminative ability. The ABIM-MEF appears to be reliable and valid. Further, factor analysis results support the ABIM’s movement to simplify the monthly evaluation form to the new Accreditation Council for Graduate Medical Education core competencies.