External validation of the improving partial risk adjustment in surgery (PRAIS-2) model for 30-day mortality after paediatric cardiac surgery
Open Access
- 1 January 2020
- Vol. 10 (11), e039236
- https://doi.org/10.1136/bmjopen-2020-039236
Abstract
Objective Independent temporal external validation of the improving partial risk adjustment in surgery model (PRAIS-2) to predict 30-day mortality in patients undergoing paediatric cardiac surgery. Design Retrospective analysis of prospectively collected data. Setting Paediatric cardiac surgery. Intervention PRAIS-2 validation was carried out using a two temporally different single centre (Bristol, UK) cohorts: Cohort 1 surgery undertaken from April 2004 to March 2009 and Cohort 2 from April 2015 to July 2019. For each subject PRAIS-2 score was calculated according to the original formula. Participants A total of 1352 (2004-2009) and 1197 (2015-2019) paediatric cardiac surgical procedures were included in the Cohort 1 and Cohort 2, respectively (median age at the procedure 6.3 and 7.1 months). Primary and secondary outcome measures PRAIS-2 performance was assessed in terms of discrimination by means of ROC (receiver operating characteristic) curve analysis and calibration by using the calibration belt method. Results PRAIS-2 score showed excellent discrimination for both cohorts (AUC 0.72 (95%CI: 0.65 to 0.80) and 0.88 (95%CI: 0.82 to 0.93), respectively). While PRAIS-2 was only marginally calibrated in Cohort 1, with a tendency to underestimate risk in lowrisk and overestimate risk in high risk procedures (P-value = 0.033), validation in Cohort 2 showed good calibration with the 95% confidence belt containing the bisector for predicted mortality (P-value = 0.143). We also observed good prediction accuracy in the non-elective procedures (N = 483;AUC 0.78 (95%CI 0.68 to 0.87); Calibration belt containing the bisector (P-value=0.589). Conclusions In a single centre UK-based cohort, PRAIS-2 showed excellent discrimination and calibration in predicting 30-day mortality in paediatric cardiac surgery including in those undergoing non-elective procedures. Our results support a wider adoption of PRAIS-2 score in the clinical practice.Keywords
Funding Information
- National Institute of Health Research Senior Investigator (NF-0616-10102)
- British Heart Foundation Accelerator Award (AA/18/7/34219)
- University of Bristol and the UK Medical Research Council (MC_UU_00011/6)
- Bristol National Institute of Health Research Biomedical Research Centre
This publication has 30 references indexed in Scilit:
- External validation of multivariable prediction models: a systematic review of methodological conduct and reportingBMC Medical Research Methodology, 2014
- A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomesStatistics in Medicine, 2014
- A goodness‐of‐fit test for the proportional odds regression modelStatistics in Medicine, 2012
- Birth Prevalence of Congenital Heart Disease WorldwideJournal of the American College of Cardiology, 2011
- Assessing the Performance of Prediction ModelsEpidemiology, 2010
- An empirically based tool for analyzing mortality associated with congenital heart surgeryThe Journal of Thoracic and Cardiovascular Surgery, 2009
- Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC CurveClinical Chemistry, 2008
- External validation is necessary in prediction research:Journal of Clinical Epidemiology, 2003
- Measuring the accuracy of prognostic judgments in oncologyJournal of Clinical Epidemiology, 1997
- The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiology, 1982