Validity Evidence for an OSCE to Assess Competency in Systems-Based Practice and Practice-Based Learning and Improvement: A Preliminary Investigation
- 1 August 2008
- journal article
- research article
- Published by Wolters Kluwer Health in Academic Medicine
- Vol. 83 (8), 775-780
- https://doi.org/10.1097/acm.0b013e31817ec873
Abstract
To determine the psychometric properties and validity of an OSCE to assess the competencies of Practice-Based Learning and Improvement (PBLI) and Systems-Based Practice (SBP) in graduate medical education. An eight-station OSCE was piloted at the end of a three-week Quality Improvement elective for nine preventive medicine and endocrinology fellows at Mayo Clinic. The stations assessed performance in quality measurement, root cause analysis, evidence-based medicine, insurance systems, team collaboration, prescription errors, Nolan's model, and negotiation. Fellows' performance in each of the stations was assessed by three faculty experts using checklists and a five-point global competency scale. A modified Angoff procedure was used to set standards. Evidence for the OSCE's validity, feasibility, and acceptability was gathered. Evidence for content and response process validity was judged as excellent by institutional content experts. Interrater reliability of scores ranged from 0.85 to 1 for most stations. Interstation correlation coefficients ranged from -0.62 to 0.99, reflecting case specificity. Implementation cost was approximately $255 per fellow. All faculty members agreed that the OSCE was realistic and capable of providing accurate assessments. The OSCE provides an opportunity to systematically sample the different subdomains of Quality Improvement. Furthermore, the OSCE provides an opportunity for the demonstration of skills rather than the testing of knowledge alone, thus making it a potentially powerful assessment tool for SBP and PBLI. The study OSCE was well suited to assess SBP and PBLI. The evidence gathered through this study lays the foundation for future validation work.Keywords
This publication has 26 references indexed in Scilit:
- An Experiential Interdisciplinary Quality Improvement Education InitiativeAmerican Journal of Medical Quality, 2006
- Techniques for measuring clinical competence: objective structured clinical examinationsMedical Education, 2004
- “The Mirror” and “The Village”: A New Method for Teaching Practice-Based Learning and Improvement and Systems-Based PracticeAcademic Medicine, 2004
- OSCE! Variations on a theme by HardenMedical Education, 2003
- Do Peer Chart Audits Improve Residents??? Performance in Providing Preventive Care?Academic Medicine, 2003
- Introducing Practice-Based Learning and Improvement ACGME Core Competencies into a Family Medicine Residency CurriculumThe Joint Commission Journal on Quality and Safety, 2003
- Development and Implementation of an Objective Structured Teaching Exercise (OSTE) to Evaluate Improvement in Feedback Skills Following a Faculty Development WorkshopTeaching and Learning in Medicine, 2003
- Itʼs Time to Focus on the Quality of GMEAcademic Medicine, 2003
- A Computer-based OSCE Station to Measure Competence in Evidence-based Medicine Skills in Medical StudentsAcademic Medicine, 2002
- Defining and Assessing Professional CompetenceJAMA, 2002