The Neglected Problem of Measurement Error in Categorical Data

Abstract

The problems created by measurement error are entirely ignored in the vast majority of statistical analyses. To adjust for the effects of measurement error requires both a theory, or model, of measurement and estimates of the relevant measurement parameters (e.g., reliability coefficients). A fairly well-developed measurement theory for interval level data has been known for quite some time. A corresponding measurement theory for categorical data is not widely known even though such data are at least as important in the social sciences as interval data. Nevertheless, such a theory exists in the statistical journals. The primary purpose of this article is pedagogical: that is, to present the foundation of this theory for binary variables, the simplest type of categorical variable, and to demonstrate that the consequences of measurement errors in binary data are different from and probably more serious than the effects of measurement errors in interval level data. The principal reason for this is that measurement errors in a binary variable are likely to have a nonzero mean and will always be negatively correlated with the underlying true scores. The former has the effect of biasing the sample estimate of the mean, often to such a degree that the likelihood that a 95% confidence interval will contain the population mean is almost nil.