Consensus Among Judges as a Function of Amount of Information

Abstract

The present research was designed to study whether amount of information and kind of information available to the judges affect the consensus among judges. Two experiments were performed, where the consensus was analyzed by the lens model equation, which has been related to the non-criterion situation by Naylor and Schenck (1966). The amount of information (two, four, and six tests) was found to have an effect on the consensus, whereas type of information available (access to the validities of the tests and their intercorrelations or not) did not have a significant effect on the consensus. The results imply that consensus among pairs of judges decreases when number of tests increase. However, the difference between four and six tests was not significant in one of the experiments. The changes in consensus as the number of tests increased was mainly a function of a decreased matching between a pair of judges with respect to the linear aspects of their judgment processes. A further analysis of the linear aspects of the judges' judgment processes indicated that policy similarity contributed more to consensus than policy consistency. The contribution of the nonlinear aspects of the judgment process to consensus was negligible in these experiments.