Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment
Top Cited Papers
- 10 August 2011
- journal article
- research article
- Published by Taylor & Francis in International Journal of Remote Sensing
- Vol. 32 (15), 4407-4429
- https://doi.org/10.1080/01431161.2011.552923
Abstract
The family of Kappa indices of agreement claim to compare a map's observed classification accuracy relative to the expected accuracy of baseline maps that can have two types of randomness: (1) random distribution of the quantity of each category and (2) random spatial allocation of the categories. Use of the Kappa indices has become part of the culture in remote sensing and other fields. This article examines five different Kappa indices, some of which were derived by the first author in 2000. We expose the indices' properties mathematically and illustrate their limitations graphically, with emphasis on Kappa's use of randomness as a baseline, and the often-ignored conversion from an observed sample matrix to the estimated population matrix. This article concludes that these Kappa indices are useless, misleading and/or flawed for the practical applications in remote sensing that we have seen. After more than a decade of working with these indices, we recommend that the profession abandon the use of Kappa indices for purposes of accuracy assessment and map comparison, and instead summarize the cross-tabulation matrix with two much simpler summary parameters: quantity disagreement and allocation disagreement. This article shows how to compute these two parameters using examples taken from peer-reviewed literature.Keywords
This publication has 37 references indexed in Scilit:
- Harshness in image classification accuracy assessmentInternational Journal of Remote Sensing, 2008
- Assessing the accuracy of species distribution models: prevalence, kappa and the true skill statistic (TSS)Journal of Applied Ecology, 2006
- Thematic Map ComparisonPhotogrammetric Engineering & Remote Sensing, 2004
- The Kappa Statistic: A Second LookComputational Linguistics, 2004
- Status of land cover classification accuracy assessmentRemote Sensing of Environment, 2001
- A review of methods for the assessment of prediction errors in conservation presence/absence modelsEnvironmental Conservation, 1997
- A review of assessing the accuracy of classifications of remotely sensed dataRemote Sensing of Environment, 1991
- Maximum Likelihood Estimation of Agreement in the Constant Predictive Probability Model, and Its Relation to Cohen's KappaBiometrics, 1990
- Coefficient Kappa: Some Uses, Misuses, and AlternativesEducational and Psychological Measurement, 1981
- Measures of Association for Cross ClassificationsJournal of the American Statistical Association, 1954