Design Sensitivity and Efficiency in Observational Studies

Abstract

An observational study attempts to draw inferences about the effects caused by a treatment when subjects are not randomly assigned to treatment or control as they would be in a randomized trial. After adjustments have been made for imbalances in measured covariates, the key source of uncertainty in an observational study is the possibility that subjects were not comparable prior to treatment in terms of some unmeasured covariate, so that differing outcomes in treated and control groups are not effects caused by the treatment. A sensitivity analysis asks about the magnitude of the departure from random assignment needed to alter the qualitative conclusions of the study, and the power of a sensitivity analysis and the design sensitivity anticipate the outcome of a sensitivity analysis under an assumed model for treatment effect. Lacking theoretical guidance, we tend to select statistical methods for use in observational studies based on their efficiency in randomized experiments. This turns out to be a mistake. A highly efficient method for detecting small treatment effects in randomized experiments need not, and often does not, have the highest power in a sensitivity analysis or the largest design sensitivity. That is, the best procedure assuming the observational study is a randomized experiment need not be the best procedure under more realistic assumptions. Small effects are sensitive to small biases, and methods targeted at detecting small effects in the absence of bias may not give due emphasis to evidence that the effect is stable and not small, and hence not easily attributed to small or moderate biases. The issue is illustrated in a practical example, is explored informally using a graphical heuristic, is studied asymptotically by determining the design sensitivity for a signed rank statistic with general scores, and is evaluated in a simulation. Among robust procedures with similar Pitman efficiencies for several distributions, there are large differences in design sensitivity and hence substantial differences in the power of a sensitivity analysis.