Using Generalizability Theory in Early Childhood Special Education

Abstract

Using a set of hypothetical data, four approaches to the estimation of interrater reliability are illustrated and compared: correlation, comparison of means, percentage of agreement, and generalizability (G) theory techniques. For these data—composed of ratings for 10 subjects on 6 items by 2 raters—the reliability estimates varied widely. The highest estimate was .90 (the simple correlation between the two raters' total scores), and the lowest was a percentage of agreement between the two raters of 26.67% (when “agreement" meant exact match). The G theory techniques allow the researcher to estimate the amount of variance attributable to multiple sources of error within one study. Although more complex than the other methods illustrated here, the G theory procedures yield very beneficial information in many reliability-estimation situations in early childhood special education.