The Reliability of Dichotomous Judgments: Unequal Numbers of Judges per Subject
- 1 October 1979
- journal article
- Published by SAGE Publications in Applied Psychological Measurement
- Vol. 3 (4), 537-542
- https://doi.org/10.1177/014662167900300410
Abstract
Consider a reliability study in which different subjects are judged on a dichotomous trait by dif ferent sets of judges, possibly unequal in number. A kappa-like measure of reliability is proposed, its correspondence to an intraclass correlation co efficient is pointed out, and a test for its statistical significance is presented. A numerical example is given.Keywords
This publication has 11 references indexed in Scilit:
- Large sample variance of kappa in the case of different sets of raters.Psychological Bulletin, 1979
- A One-Way Components of Variance Model for Categorical DataBiometrics, 1977
- The Measurement of Observer Agreement for Categorical DataBiometrics, 1977
- Axymptotic normality of X/sup 2/ in mxn tables with n large and small cell expectationsPublished by Office of Scientific and Technical Information (OSTI) ,1977
- Measuring Agreement between Two Judges on the Presence or Absence of a TraitBiometrics, 1975
- The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of ReliabilityEducational and Psychological Measurement, 1973
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971
- Measures of response agreement for qualitative data: Some generalizations and alternatives.Psychological Bulletin, 1971
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960