A Computer Program to Calculate Adjusted and Unadjusted Interrater Reliabilities for Sets and Subsets of Judges

Abstract

A computer program to be used to assess interrater reliabilities has been written. Given judges' ratings on a set of variables pertaining to subjects or events, the program will produce for each variable, a printed analysis of variance summary table for between subjects/ events, within subjects/events, and between judges sources of variability, a reliability coefficient, an adjusted reliability coefficient and means and standard deviations for each rater. Further, analysis of variance and reliability output for subsets of judges is generated.