An Investigation of the Robustness of the t Test to Unit of Analysis Violations

Abstract

In the first part of this study, sampling experiments were used to assess the Type I error rates of the t test in situations where classes or schools were randomly assigned to groups but analyses were carried out on individual student scores. It was found that Type I error rates were grossly inflated under all conditions investigated. In the second part of the study, mathematical analysis was used to demonstrate the fact that even small amounts of between class (or school) variation can cause large inflations in the Type I error rate of the t test when analyses are incorrectly carried out on individual student scores.