Answering the Call for a Standard Reliability Measure for Coding Data
Top Cited Papers
- 1 April 2007
- journal article
- research article
- Published by Taylor & Francis in Communication Methods and Measures
- Vol. 1 (1), 77-89
- https://doi.org/10.1080/19312450709336664
Abstract
In content analysis and similar methods, data are typically generated by trained human observers who record or transcribe textual, pictorial, or audible matter in terms suitable for analysis. Conclusions from such data can be trusted only after demonstrating their reliability. Unfortunately, the content analysis literature is full of proposals for so-called reliability coefficients, leaving investigators easily confused, not knowing which to choose. After describing the criteria for a good measure of reliability, we propose Krippendorff's alpha as the standard reliability measure. It is general in that it can be used regardless of the number of observers, levels of measurement, sample sizes, and presence or absence of missing data. To facilitate the adoption of this recommendation, we describe a freely available macro written for SPSS and SAS to calculate Krippendorff's alpha and illustrate its use with a simple example.Keywords
This publication has 19 references indexed in Scilit:
- Reliability in Content Analysis.Human Communication Research, 2004
- Statistical Methods for Rates and ProportionsWiley Series in Probability and Statistics, 2003
- Content Analysis in Mass Communication: Assessment and Reporting of Intercoder ReliabilityHuman Communication Research, 2002
- Coefficient Kappa: Some Uses, Misuses, and AlternativesEducational and Psychological Measurement, 1981
- On Generalizations Of The G Index And The Phi Coefficient To Nominal ScalesMultivariate Behavioral Research, 1979
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971
- Estimating the Reliability, Systematic Error and Random Error of Interval DataEducational and Psychological Measurement, 1970
- A Note on the G Index of AgreementEducational and Psychological Measurement, 1964
- Communications Through Limited Response QuestioningPublic Opinion Quarterly, 1954
- Coefficient alpha and the internal structure of testsPsychometrika, 1951