基于分类和回归树决策树的网络大数据集离群点动态检测算法

doi:10.13229/j.cnki.jdxbgxb.20220434

Abstract

Abstract:

There are massive data in big data sets， and when the data scale expands to a certain extent， the processing efficiency of discrete point detection is limited. Therefore， a dynamic outlier detection algorithm based on CART decision tree was proposed. Firstly， the abnormal data standard of large data set was divided， the data dispersion degree by variance was measured， the abnormal data sample association rule matrix by support vector machine was established， the abnormal data range of large data set was clarified， and the amount of outlier detection calculation by dynamic meshing strategy was reduced. Then， the classification and regression trees（CART） decision tree method was used to take Boolean detection at the branch nodes， unify the data to be detected as continuous data， arrange the training data set in ascending order， calculate the maximum information gain of the data， prune the decision tree until no non leaf nodes can be replaced， and obtain the dynamic detection results of outliers. Simulation results show that the proposed algorithm has high outlier detection accuracy， short detection time， significant computational advantages， and can provide positive help for the reliable application of large data sets.

Key words: classification and regression trees（CART） decision tree, large data sets, outlier detection, data preprocessing, meshing, Gini coefficient

CLC Number:

TP393

Li-fang FU,Zhuo CHEN,Chang-lin AO. Dynamic outlier detection algorithm for network large data set based on classification and regression trees decision tree[J].Journal of Jilin University(Engineering and Technology Edition), 2023, 53(9): 2620-2625.

Figures/Tables 5

Table 1

Table 2

Fig.1

Fig.2

Fig.3

References 20

1	杨晓玲, 冯山, 袁钟. 基于相对距离的反k近邻树离群点检测[J]. 电子学报, 2020, 48(5): 937-945.
	Yang Xiao-ling, Feng shan, Yuan Zhong. Outlier detection based on reversed k-nearest neighborhood mst of relative distance measure[J]. Acta Electronica Sinica, 2020, 48(5): 937-945.
2	Vafaei N, Ribeiro R A, Camarinha-Matos L M. Comparison of normalization techniques on data sets with outliers[J]. International Journal of Decision Support System Technology, 2022, 14(1): 1-17.
3	张倩倩, 于炯, 李梓杨, 等. 基于近邻传播的离群点检测算法[J]. 计算机应用研究, 2021, 38(6): 1662-1667.
	Zhang Qian-qian, Yu Jiong, Li Zi-yang, et al. Outlier detection algorithm based on affinity propagation[J]. Application Research of Computers, 2021, 38(6): 1662-1667.
4	江峰, 王凯郦, 于旭, 等. 基于粗糙熵的离群点检测方法及其在无监督入侵检测中的应用[J]. 控制与决策, 2020, 35(5): 1199-1204.
	Jiang Feng, Wang Kai-li, Yu Xu, et al. A rough entropy-based approach to outlier detection and its application in unsupervised intrusion detection[J]. Control and Decision, 2020, 35(5): 1199-1204.
5	Belhadi A, Djenouri Y, Djenouri D, et al. Deep learning versus traditional solutions for group trajectory outliers[J]. IEEE Transactions on Cybernetics, 2020, 52(6): 4508-4519.
6	袁庆军, 王安, 王永娟, 等. 基于流形学习能量数据预处理的模板攻击优化方法[J]. 电子与信息学报, 2020, 42(8): 1853-1861.
	Yuan Qing-jun, Wang An, Wang Yong-juan, et al. An improved template analysis method based on power traces preprocessing with manifold learning[J]. Journal of Electronics & Information Technology, 2020, 42(8): 1853-1861.
7	Ghani M U, Rafi M, Tahir M A. Discriminative adaptive sets for multi-label classification[J]. IEEE Access, 2020, 8: 227579-227595.
8	邓泓, 刘志超, 彭莹琼, 等. 基于Fibonacci采样的数据预处理方法研究[J]. 江西师范大学学报: 自然科学版, 2021, 45(1): 60-66.
	Deng Hong, Liu Zhi-chao, Peng Ying-qiong, et al. The study on data preprocessing method based on fibonacci sampling[J]. Journal of Jiangxi Normal University (Natural Sciences Edition), 2021, 45(1): 60-66.
9	Sripriya T P, Srinivasan M R, Gallo M. Robust distance measure to detect outliers for categorical data[J]. Soft Computing, 2020, 24(18): 1-8.
10	Li N, Zhao X W, Mu H L, et al. Research on the self-repairing model of outliers in energy data based on regional convergence[J]. Energies, 2020, 13(18): No.4909.
11	刘云, 郑文凤, 张轶. 模糊残差算法对离群点数据的优化研究[J].小型微型计算机系统, 2021, 42(6): 1321-1326.
	Liu Yun, Zheng Wen-feng, Zhang Yi. Optimization of outlier data by fuzzy residual algorithm[J]. Journal of Chinese Computer Systems, 2021, 42(6): 1321-1326.
12	王习特, 朱宗梅, 于雪苹, 等. 异构分布式环境中的并行离群点检测算法[J]. 湖南大学学报: 自然科学版, 2020, 47(10): 100-110.
	Wang Xi-te, Zhu Zong-mei, Yu Xue-ping, et al. Parallel outlier detection algorithm in heterogeneous distributed environment[J]. Journal of Hunan University (Natural Sciences), 2020, 47(10): 100-110.
13	Yang L, Lu Y Z, Yang S X, et al. An evolutionary game based secure clustering protocol with fuzzy trust evaluation and outlier detection for wireless sensor networks[J]. IEEE Sensors Journal, 2021, 21(12): 13935-13947.
14	水泽农, 张星宇, 沙朝锋. 基于最优输运和k-近邻的离群文档检测[J]. 计算机科学, 2021, 48(7): 105-111.
	Shui Ze-nong, Zhang Xing-yu, Sha Chao-feng. Outlier document detection via optimal transport and k-nearest neighbor[J]. Computer Science, 2021, 48(7): 105-111.
15	Yu K Q, Shi W, Santoro N. Designing a streaming algorithm for outlier detection in data mining—an incrementa approach[J]. Sensors, 2020, 20(5): No.1261.
16	Hagan R, Langston M A. Molecular subtyping and outlier detection in human disease using the paraclique algorithm[J]. Algorithms, 2021, 14(2): No.63.
17	林雪. 海量不确定数据集中离群点快速检测方法仿真[J]. 计算机仿真, 2021, 38(6): 378-382.
	Lin Xue. Simulation of quick detection method for outliers in massive uncertain data sets[J]. Computer Simulation, 2021,38(6): 378-382.
18	Mouret F, Albughdadi M, Duthoit S, et al. Outlier detection at the parcel-level in wheat and rapeseed crops using multispectral and sar time series[J]. Remote Sensing, 2021, 13(5): No.956.
19	董泽, 贾昊. 基于EWT-LOF的热工过程数据异常值检测方法[J]. 仪器仪表学报, 2020, 41(2): 126-134.
	Dong Ze, Jia Hao. Outlier detection method for thermal process data based on EWT-LOF[J]. Chinese Journal of Scientific Instrument, 2020, 41(2): 126-134.
20	Riahi-Madvar M, Azirani A A, Nasersharif B, et al. A new density-based subspace selection method using mutual information for high dimensional outlier detection[J]. Knowledge-Based Systems, 2021, 216(2): No.106733.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 10

[1]	LI Shoutao, LI Yuanchun. Autonomous Mobile Robot Control Algorithm Based on Hierarchical Fuzzy Behaviors in Unknown Environments[J]. 吉林大学学报(工学版), 2005, 35(04): 391 -397 .
[2]	Li Hong-ying; Shi Wei-guang;Gan Shu-cai. Electromagnetic properties and microwave absorbing property of Z type hexaferrite Ba_3-xLa_xCo₂Fe₂₄O₄₁[J]. 吉林大学学报(工学版), 2006, 36(06): 856 -0860 .
[3]	Zhang Quan-fa，Li Ming-zhe，Sun Gang，Ge Xin . Comparison between flexible and rigid blank-holding in multi-point forming[J]. 吉林大学学报(工学版), 2007, 37(01): 25 -30 .
[4]	Yang Shu-kai, Song Chuan-xue, An Xiao-juan, Cai Zhang-lin . Analyzing effects of suspension bushing elasticity on vehicle yaw response character with virtual prototype method[J]. 吉林大学学报(工学版), 2007, 37(05): 994 -0999 .
[5]	. [J]. 吉林大学学报(工学版), 2007, 37(06): 1284 -1287 .
[6]	Liu Han-bing, Jiao Yu-ling, Liang Chun-yu,Qin Wei-jun . Effect of shape function on computing precision in meshless methods[J]. 吉林大学学报(工学版), 2007, 37(03): 715 -0720 .
[7]	Yang Qing-fang, Chen Lin . Division approach of traffic control work zone[J]. 吉林大学学报(工学版), 2006, 36(增刊2): 139 -142 .
[8]	Li Yue-ying，Liu Yong-bing，Chen Hua . Surface hardening and tribological properties of a cam materials[J]. 吉林大学学报(工学版), 2007, 37(05): 1064 -1068 .
[9]	Zhang He-sheng, Zhang Yi, Wen Hui-min, Hu Dong-cheng . Estimation approaches of average link travel time using GPS data[J]. 吉林大学学报(工学版), 2007, 37(03): 533 -0537 .
[10]	Qu Zhao-wei，Chen Hong-yan，Li Zhi-hui，Hu Hong-yu，Wei Wei . 2D view reconstruction method based on single calibration pattern [J]. 吉林大学学报(工学版), 2007, 37(05): 1159 -1163 .

序号	类型	潜在表现	分析结果
1	数据失效	无法完成解析任务	无需过滤
1	数据失效	不在有效范围内	无需过滤
2	数据跳变	数据产生大幅度改变	无需过滤
		数据发生改变后随即恢复正常	无需过滤
		数据改变幅度极小	需要过滤
3	其他	数据状态稳定	需要过滤

样本	数据量/万条	样本大小/MB
A	30	894
B	40	1056
C	50	1719
D	60	2493

Dynamic outlier detection algorithm for network large data set based on classification and regression trees decision tree

RICH HTML

PDF (PC)

Abstract

Cite this article

share this article

Figures/Tables 5

References 20

Related Articles 3

Metrics

Comments

Recommended 10

[1]	Guo-hui CHEN,Ye-yin XU,Ying-hou JIAO. Meshing stiffness calculation and vibration analysis of helical gear considering deflection [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(7): 1902-1910.
[2]	CAI Zhong-chang, LIU Hui, XIANG Chang-le, ZHANG Xi-qing, WANG Ming-zheng. Characteristics of forced torsional vibration and dynamic load for vehicle multistage planetary transmission [J]. 吉林大学学报(工学版), 2012, 42(01): 19-26.
[3]	WANG Zi-qiang，DUAN Ai-ling，ZHANG De-xian . Efficient outlier detection algorithm based on support vector data description [J]. 吉林大学学报(工学版), 2009, 39(02): 499-0503.