Toward Meaningful Evaluation of Clinical Competence: The Role of Direct Observation in Clerkship Ratings
Open Access
- 1 October 2004
- journal article
- research article
- Published by Wolters Kluwer Health in Academic Medicine
- Vol. 79 (10), S21-S24
- https://doi.org/10.1097/00001888-200410001-00007
Abstract
The lack of direct observation by faculty may affect meaningful judgments of clinical competence. The purpose of this study was to explore the influence of direct observation on reliability and validity evidence for family medicine clerkship ratings of clinical performance. Preceptors rating family medicine clerks (n = 172) on a 16-item evaluation instrument noted the data-source for each rating: note review, case discussion, and/or direct observation. Mean data-source scores were computed and categorized as low, medium or high, with the high-score group including the most direct observation. Analyses examined the influence of data-source on interrater agreement and associations between clerkship clinical scores (CCS) and scores from the National Board of Medical Examiners (NBME®) subject examination as well as a fourth-year standardized patient-based clinical competence examination (M4CCE). Interrater reliability increased as a function of data-source; for the low, medium, and high groups, intraclass correlation coefficients were .29, .50, and .74, respectively. For the high-score group, there were significant positive correlations between CCS and NBME score (r = .311, p = .054); and between CCS and M4CCE (r = .423, p = .009). Reliability and validity evidence for clinical competence is enhanced when more direct observation is included as a basis for clerkship ratings.Keywords
This publication has 19 references indexed in Scilit:
- Faculty and the Observation of Trainees’ Clinical Skills: Problems and OpportunitiesAcademic Medicine, 2004
- SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance RatingsTeaching and Learning in Medicine, 2003
- Physician Clinical Performance AssessmentJAMA, 2003
- Validity: on the meaningful interpretation of assessment dataMedical Education, 2003
- Effectiveness of a focused educational intervention on resident evaluations from facultyJournal of General Internal Medicine, 2001
- Shortcomings in the evaluation of studentsʼ clinical skills and behaviors in medical schoolAcademic Medicine, 1999
- The Correction for AttenuationEducational and Psychological Measurement, 1996
- Assessment of Clinical Skills of Residents Utilizing Standardized PatientsAnnals of Internal Medicine, 1991
- Evaluating a direct observation exercise in a psychiatric clerkshipMedical Education, 1984
- The Deficiencies of the Case Presentation as a Method of Clinical TeachingNew England Journal of Medicine, 1971