Using Localization Data from Image Interpretations to Improve Estimates of Performance Accuracy

Abstract

A recently developed model uses the localization of abnormalities on images to improve statistical precision in measuring detection accuracy A_z, the area below an observer's receiver operating characteristic (ROC) curve for ratings of sampled normal and abnormal cases. This study evaluated that improvement by investigating how much the standard error of estimated A_z decreased when the statistical analysis included localization data. Comparisons of analyses with vs without localizations were made for: 1) the estimates of A_z from observers' rating ROC curves for nodular lesions on clinical chest films and liver CT scans; 2) the probability of correct choices between paired samples of normal and abnormal cases (equivalent to A_z); and 3) the sampling distributions of A_z measured in Monte Carlo simulations of 2,000 independent rating experiments. Localization information considerably improved the precision of A_z estimates, particularly when detection accuracy was low ( A_z ~ 0.60). These data provided roughly the same benefits in estimation precision as would two-to-fourfold increases in the sizes of both 1) the samples of positive and negative cases and 2) the observer samples used to estimate A_z means. Key words: ROC analysis; LROC; observer performance accuracy; observer vanability; detection; localization; image interpretation. (Med Decis Making 2000;20:170-185)