A proposal for interpreting and reporting negative studies

Abstract

An issue of continuing interest is the interpretation and reporting of ‘negative’ studies, namely studies that do not find statistically significant differences. The most common approach is the design-power method which determines, irrespective of the observed difference, what differences the study could have been expected to detect. We propose an alternative approach, the application of equivalence testing methods, where we define equivalence to mean that the actual difference lies within some specified limits. This approach, in contrast to the design-power approach, provides a way of quantifying (with p-values) what was actually determined from the study instead of saying what the study may or may not have accomplished with some degree of certainty (power). For example, a possible outcome of the equivalence testing approach is the conclusion at the 5 per cent level that two means (or proportions) do not differ by more than some specified amount. The equivalence testing approach applies to any study design. We illustrate the method with a cancer clinical trial and an epidemiologic case-control study. In addition, for those studies in which one cannot specify limits a priori, we propose the use of equivalence curves to summarize and present the study results.