A Study on the Relationships of Classifier Performance Metrics
- 1 November 2009
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10823409,p. 59-66
- https://doi.org/10.1109/ictai.2009.25
Abstract
There is no general consensus on which classifier performance metrics are better to use as compared to others. While some studies investigate a handful of such metrics in a comparative fashion, an evaluation of specific relationships among a large set of commonly-used performance metrics is much needed in the data mining and machine learning community. This study provides a unique insight into the underlying relationships among classifier performance metrics. We do so with a large case study involving 35 datasets from various domains and the C4.5 decision tree algorithm. A common property of the 35 datasets is that they suffer from the class imbalance problem. Our approach is based on applying factor analysis to the classifier performance space which is characterized by 22 performance metrics. It is shown that such a large number of performance metrics can be grouped into two-to-four relationship-based groups extracted by factor analysis. This work is a step in the direction of providing the analyst with an improved understanding about the different relationships and groupings among the performance metrics, thus facilitating the selection of performance metrics that capture relatively independent aspects of a classifier's performance.Keywords
This publication has 12 references indexed in Scilit:
- An Empirical Study of Learning from Imbalanced Data Using Random ForestPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2007
- Experimental perspectives on learning from imbalanced dataPublished by Association for Computing Machinery (ACM) ,2007
- The relationship between Precision-Recall and ROC curvesPublished by Association for Computing Machinery (ACM) ,2006
- Comparative Assessment of Software Quality Classification Techniques: An Empirical Case StudyEmpirical Software Engineering, 2004
- Data mining in metric spacePublished by Association for Computing Machinery (ACM) ,2004
- A case study of applying boosting naive bayes to claim fraud diagnosisIEEE Transactions on Knowledge and Data Engineering, 2004
- Software quality prediction using median-adjusted class labelsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2003
- Machine Learning for the Detection of Oil Spills in Satellite Radar ImagesMachine Learning, 1998
- Statistical Classification Methods in Consumer Credit Scoring: A ReviewJournal of the Royal Statistical Society Series A: Statistics in Society, 1997
- Measuring dynamic program complexityIEEE Software, 1992