External Validity of Risk Models: Use of Benchmark Values to Disentangle a Case-Mix Effect From Incorrect Coefficients
Open Access
- 31 August 2010
- journal article
- research article
- Published by Oxford University Press (OUP) in American Journal of Epidemiology
- Vol. 172 (8), 971-980
- https://doi.org/10.1093/aje/kwq223
Abstract
Various performance measures related to calibration and discrimination are available for the assessment of risk models. When the validity of a risk model is assessed in a new population, estimates of the model's performance can be influenced in several ways. The regression coefficients can be incorrect, which indeed results in an invalid model. However, the distribution of patient characteristics (case mix) may also influence the performance of the model. Here the authors consider a number of typical situations that can be encountered in external validation studies. Theoretical relations between differences in development and validation samples and performance measures are studied by simulation. Benchmark values for the performance measures are proposed to disentangle a case-mix effect from incorrect regression coefficients, when interpreting the model's estimated performance in validation samples. The authors demonstrate the use of the benchmark values using data on traumatic brain injury obtained from the International Tirilazad Trial and the North American Tirilazad Trial (1991–1994).Keywords
This publication has 34 references indexed in Scilit:
- Assessing the Performance of Prediction ModelsEpidemiology, 2010
- Validation of Biomarker-Based Risk Prediction ModelsClinical Cancer Research, 2008
- Predicting Outcome after Traumatic Brain Injury: Development and International Validation of Prognostic Scores Based on Admission CharacteristicsPLoS Medicine, 2008
- The Performance of Risk Prediction ModelsBiometrical Journal, 2008
- A systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomesJournal of Clinical Epidemiology, 2008
- Internal validation of predictive modelsJournal of Clinical Epidemiology, 2001
- Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data setsStatistics in Medicine, 2000
- Projecting Individualized Probabilities of Developing Breast Cancer for White Females Who Are Being Examined AnnuallyJNCI Journal of the National Cancer Institute, 1989
- Two further applications of a model for binary regressionBiometrika, 1958
- VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITYMonthly Weather Review, 1950