Impact of different customization strategies in the performance of a general severity score

Abstract

To compare the impact of two different customization strategies in the performance of the admission Mortality Probability Model II (MPM II0) using formal statistical assessment. Analysis of the database of a multicenter, multinational, prospective cohort study, EURICUS-1, involving 89 intensive care units (ICUs) from 12 European countries. Eighty-nine ICUs from 12 European countries. Data from 16,060 patients consecutively admitted to 89 ICUs from 12 European countries were collected during a 4-month period. In accordance with original MPM II0 criteria, the following patients were excluded from analysis: patients <18 yrs of age; patients considered readmissions; patients with acute myocardial infarction or burns; and patients in the postoperative period recovering from coronary artery bypass surgery. A total of 10,397 patients were analyzed. Collection of the data necessary for the calculation of MPM II0 and basic demographic statistics. Vital status at hospital discharge was registered. Two new logistic regression equations were developed to relate MPM II sub 0 to mortallty after splitting the database into development and validation samples, the first with the original logit of MPM II0 as an independent variable (first-level customization), and the second with all 15 original variables (second-level customization). Discrimination (area under the receiver operating curve), Hosmer-Lemeshow goodness-of-fit tests H and C, and observed/expected mortality ratios were evaluated in both samples and within relevant subgroups. The discriminative capability of the models was only slightly affected by customization (0.810 vs. 0.803), remaining lower than in the original description of the MPM II0 (0.824). Calibration improved, with Hosmer-Lemeshow goodness-of-fit tests H and C showing a good fit of the models. However, the formal evaluation of discrimination, calibration, and observed/expected mortality ratios across relevant subgroups appeared to be poor in some groups. In this ICU patient database, second-level customization was more effective than first-level customization in improving the overall goodness-of-fit of MPM II0 and should probably be chosen as the preferential strategy to improve the fit of a model when the sample size is large enough. However, second level customization had only a slight impact on discrimination. Its effects on the uniformity of fit are insufficlent to overcome the problems that can arise when the model is applied in populations in which the case-mix is distinct from the population where it was originally developed. (Crit Care Med 1997; 25:2001-2008)