吉林大学学报(工学版) ›› 2012, Vol. 42 ›› Issue (02): 463-468.

Previous Articles     Next Articles

New performance evaluation method for classifier

LI Jun1,2, LI Xiong-fei1, DONG Yuan-fang1,3, ZHAO Hai-ying4   

  1. 1. Key Laboratory of Symbolic Computation and Knowledge Engineering for Ministry of Education, Jilin University, Changchun 130012, China;
    2. Department of Mathematics, Changchun University of Science and Technology, Changchun 130022, China;
    3. School of Economics and Management, Changchun University of Science and Technology, Changchun 130022, China;
    4. School of Computer Science and Technology, Xinjiang Normal University, Wulumuqi, 830000
  • Received:2011-02-11 Online:2012-03-01 Published:2012-03-01

PDF (PC)

1329

Abstract: An imbalanced classifier performance evaluation method, weighted Area Under the Curve (wAUC), is proposed to solve the evaluation problem of imbalanced or class-skewed data classifiers. This method makes use of different weights in different regions according to the values of the True Positive rate (TPrate) to focus on the accuracy of positive class when calculating the weighted area under the Receiver Operating Characteristic (ROC) curve. It is beneficial to distinguish the different contributions of the accuracies on different classes to the overall performance. The features of the weight function are discussed and the characteristics of the wAUC are analyzed. Theoretical analysis and experimental results show that the proposed wAUC method is superior to OP and AUC methods.

Key words: computer software and theory, imbalanced data, classification, performance evaluation, AUC

CLC Number: 

  • TG181
[1] Chawla N V.Data Mining for Imbalanced Datasets: an Overview.Data Mining and Knowledge Discovery Handbook ［M］,Heidelberg: Springer,2010: 875-886.

[2] Fawcett T.An introduction to ROC analysis［J］.Pattern Recognition Letters,2006,27(8): 861-874.

[3] Egan J P.Signal Detection Theory and ROC Analysis,Series in Cognition and Perception ［M］.New York:Academic Press,1975.

[4] Spackman K A.Signal detection theory: Valuable tools for evaluating inductive learning //Proc Sixth International Workshop on Machine Learning.Morgan Kaufman,San Mateo,CA,1989: 160-163.

[5] Japkowicz N,Stephen S.The class imbalance problem: a systematic study ［J］.Intelligent Data Analysis,2002,l6:40-49.

[6] Chawla N V,Japkowicz N,Kotcz A.Editorial: special issue on learning from imbalanced data sets［J］.SIGKDD Exploration Newsletters,2004,6(1):1-6.

[7] Elazmeh W,Japkowicz N,Matwin S.Evaluating misclassifications in imbalanced data //Proc 17th European Conference on Machine Learning,2006: 126-137.

[8] Huang J,Ling C X.Using AUC and accuracy in evaluating learning algorithms［J］.IEEE Trans on Knowledge and Data Engineering,2005,17:299-310.

[9] Daskalaki S,Kopanas I,Avouris N.Evaluation of classifiers for an uneven class distribution problem ［J］.Applied Artificial Intelligence,2006,20: 381-417.

[10] Kubat M,Matwin S.Adressing the curse of imbalanced training sets: one-sided selection //Proc 14th Intl Conf on Machine Learning Nashville,USA,1997:179-186.

[11] Weng C G,Poon J.A new evaluation measure for imbalanced datasets //Proc Seventh Australasian Data Mining Conference (AusDM 2008),Glenelg,Australia.2008:27-32.

[12] Ranawana R,Palade V.Optimized precision-anew measure for classifier performance evaluation //Proc IEEE Congress on Evolutionary Computation,2006:2254-2261.

[13] Efron B,Tibshirani R.An Introduction to the Bootstrap ［M］.Chapman and Hall,1993.
[1] DONG Sa, LIU Da-you, OUYANG Ruo-chuan, ZHU Yun-gang, LI Li-na. Logistic regression classification in networked data with heterophily based on second-order Markov assumption [J]. Journal of Jilin University(Engineering and Technology Edition), 2018, 48(5): 1571-1577.
[2] LIU Jie, ZHANG Ping, GAO Wan-fu. Feature selection method based on conditional relevance [J]. 吉林大学学报(工学版), 2018, 48(3): 874-881.
[3] CHEN Mian-shu, SU Yue, SANG Ai-jun, LI Pei-peng. Image classification methods based on space vector model [J]. 吉林大学学报(工学版), 2018, 48(3): 943-951.
[4] CHEN Tao, CUI Yue-han, GUO Li-min. Improved algorithm of multiple signal classification for single snapshot [J]. 吉林大学学报(工学版), 2018, 48(3): 952-956.
[5] YANG Hong-yu, XU Jin. An Android malware static detection model [J]. 吉林大学学报(工学版), 2018, 48(2): 564-570.
[6] XU Jin-kai, WANG Yu-tian, ZHANG Shi-zhong. Dynamic characteristics of a heavy duty parallel mechanism with actuation redundancy [J]. 吉林大学学报(工学版), 2017, 47(4): 1138-1143.
[7] DONG Li-yan, SUI Peng, SUN Peng, LI Yong-li. Novel naive Bayes classification algorithm based on semi-supervised learning [J]. 吉林大学学报(工学版), 2016, 46(3): 884-889.
[8] DONG Sa, LIU Da-you, LI Li-na, OUYANG Ruo-chuan, CHAI Xiao-li. Relational neighbor algorithm based on class propagation distributions for classification in networked data with heterophily [J]. 吉林大学学报(工学版), 2016, 46(2): 522-527.
[9] MA An-xiang, ZHANG Chang-sheng, ZHANG Bin, ZHANG Xiao-hong. Adaptive artificial bee colony algorithm for classification problem [J]. 吉林大学学报(工学版), 2016, 46(1): 252-258.
[10] ZHENG Xin, PENG Zhen-ming, XING Yan. Novel method of evaluating image segmentation algorithms based on activity degree [J]. 吉林大学学报(工学版), 2016, 46(1): 311-317.
[11] XIN Yu, YANG Jing, XIE Zhi-qiang. K-topic increment training algorithm based on LDA [J]. 吉林大学学报(工学版), 2015, 45(4): 1242-1252.
[12] ZHAO Dong, HAN Xiao-yan, ZHAO Hong-wei, YU Fan-hua. IOT node load balancing strategy based on classification optimization [J]. 吉林大学学报(工学版), 2015, 45(3): 926-931.
[13] SI Wei-jian, LI Xiao-lin. Low-complexity near-sources location estimation based on the idea of compression [J]. 吉林大学学报(工学版), 2015, 45(3): 991-997.
[14] QI Xing-da, LI Xian-jun, LIU Si-yu, MENG Dong-hui. Differentiation research on industrial technology innovation ability by DEA and PCA [J]. 吉林大学学报(工学版), 2015, 45(3): 1017-1023.
[15] LI Hai-jian,DONG Hong-hui,SHI Yuan-chao,JIA Li-min,GUO Wei-feng. Vehicle classification with a single magnetic sensor for urban road [J]. 吉林大学学报(工学版), 2015, 45(1): 97-103.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright © Journal of Jilin University(Engineering and Technology Edition), All Rights Reserved.
Tel: +86-431-85095297　Fax: +86-431-85094074　E-mail: xbgxb@jlu.edu.cn
Powered by Beijing Magtech Co. Ltd