基于随机森林算法的大数据异常检测模型设计

doi:10.13229/j.cnki.jdxbgxb.20220598

摘要/Abstract

摘要：

针对大数据异常检测过程易受边缘数据的干扰，导致大数据异常检测准确率较差的问题，提出了一种基于随机森林算法的大数据异常检测模型。首先，利用改进k-means算法对大数据实行聚类处理，采用主成分分析法提取大数据特征；然后，构建基于随机森林分类器的大数据异常检测模型，将提取的特征输入到模型中，构建决策树，并通过动态更新决策树的权重值提高分类器的分类精度；最后，输出分类结果，完成大数据的异常检测。实验结果表明，本文模型的检测时间约为25 s，大数据异常检测准确率平均值为91%，误报率为4.5%。

关键词: 大数据聚类, 特征提取, 主成分分析法, 随机森林分类器, 决策树, 更新权重

Abstract:

Aiming at the problem that Big data anomaly detection process is easily interfered by edge data， which leads to poor accuracy of Big data anomaly detection， a big data anomaly detection model based on Random forest algorithm was proposed. Firstly， the improved k-means algorithm was used to cluster the big data， and the principal component analysis method was used to extract the features of the big data. Then a big data anomaly detection model based on random forest classifier was built， the extracted features was inputted into the model， a decision tree was built， and the classification accuracy of the classifier was improved by dynamically updating the weight value of the decision tree. Finally， the classification results are output to complete the anomaly detection of big data. The experimental results show that the detection time of the proposed model is about 25 s， the average big data anomaly detection accuracy is 91%， and the false alarm rate is 4.5%.

Key words: big data clustering, feature extraction, principal component analysis, random forest classifier, decision tree, update weights

中图分类号:

TM714

宋世军,樊敏. 基于随机森林算法的大数据异常检测模型设计[J]. 吉林大学学报(工学版), 2023, 53(9): 2659-2665.

Shi-jun SONG,Min FAN. Design of big data anomaly detection model based on random forest algorithm[J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(9): 2659-2665.

图/表 5

图1

图2

表1

图3

图4

参考文献 15

1	刘永辉, 张显, 孙鸿雁, 等. 能源互联网背景下电力市场大数据应用探讨[J]. 电力系统自动化, 2021, 45(11): 1-10.
	Liu Yong-hui, Zhang Xian, Sun Hong-yan, et al. Discussion on application of big data in electricity market in background of energy internet[J]. Automation of Electric Power Systems, 2021, 45(11): 1-10.
2	姜丹, 梁春燕, 吴军英, 等. 基于大数据分析的电力运行数据异常检测示警方法[J]. 中国测试, 2020, 46(7): 18-23.
	Jiang Dan, Liang Chun-yan, Wu Jun-ying, et al. Alarm method of power operation data anomaly detection based on big data analysis[J]. China Measurement & Test, 2020, 46(7): 18-23.
3	万磊, 陈成, 黄文杰, 等. 基于BRB和LSTM网络的电力大数据用电异常检测方法[J]. 电力建设, 2021, 42(8): 38-45.
	Wan Lei, Chen Cheng, Huang Wen-jie, et al. Power abnormity detection method based on power big data applying BRB and LSTM network[J]. Electric Power Construction, 2021, 42(8): 38-45.
4	李清. 基于改进PSO-PFCM聚类算法的电力大数据异常检测方法[J]. 电力系统保护与控制, 2021, 49(18): 161-166.
	Li Qing. Power big data anomaly detection method based on an improved PSO-PFCM clustering algorithm[J]. Power System Protection and Control, 2021, 49(18): 161-166.
5	丁小欧, 于晟健, 王沐贤, 等. 基于相关性分析的工业时序数据异常检测[J]. 软件学报, 2020, 31(3): 726-747.
	Ding Xiao-ou, Yu Sheng-jian, Wang Mu-xian, et al. Anomaly detection on industrial time series based on correlation analysis[J]. Journal of Software, 2020, 31(3): 726-747.
6	谢桦, 陈昊, 邓晓洋, 等. 基于改进k-means聚类技术与半不变量法的电-气综合能源系统运行风险评估方法[J]. 中国电机工程学报, 2020, 40(1): 59-69, 374.
	Xie Hua, Chen Hao, Deng Xiao-yang, et al. Electric-gas integrated energy system operational risk assessment based on improved k-means clustering technology and semi-invariant method[J]. Proceedings of the CSEE, 2020, 40(1): 59-69, 374.
7	吴金蔚. φ-混合样本下密度函数在有限点处的联合渐近分布[J]. 信阳师范学院学报: 自然科学版, 2021, 34(4): 541-544.
	Wu Jin-wei. The joint asymptotic distribution of probability density function in a finite number of points under φ-mixing samples[J]. Journal of Xinyang Normal University (Natural Science Edition), 2021, 34(4): 541-544.
8	张重远, 胡焕, 程槐号, 等. 基于欧氏距离分析的电力变压器绕组变形程度与类型的诊断方法[J]. 高压电器, 2020, 56(1): 224-230.
	Zhang Zhong-yuan, Hu Huan, Cheng Huai-hao, et al. Diagnostic method to determine degree and type of winding deformation in power transformer based on euclidean distance[J]. High Voltage Apparatus, 2020, 56(1): 224-230.
9	代瑾, 陈莹. 联合线性判别和图正则的任务导向型跨模态检索[J]. 计算机辅助设计与图形学学报, 2021, 33(1): 106-115.
	Dai Jin, Chen Ying. Joint Linear Discrimination and graph regularization for task-oriented cross-modal retrieval[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(1): 106-115.
10	蔡瑞初, 李嘉豪, 郝志峰. 基于类内最大均值差异的无监督领域自适应算法[J]. 计算机应用研究, 2020, 37(8): 2371-2375.
	Cai Rui-chu, Li Jia-hao, Hao Zhi-feng.Unsupervised domain adaptive algorithm with intra-class maximum mean discrepancy[J]. Application Research of Computers, 2020, 37(8): 2371-2375.
11	胡善科, 秦玉华, 段如敏, 等. 联合矩阵局部保持投影的近红外光谱特征提取[J]. 光谱学与光谱分析, 2020, 40(12): 3772-3777.
	Hu Shan-ke, Qin Yu-hua, Duan Ru-min, et al. Research on feature extraction of near-infrared spectroscopy based on joint matrix local preserving projection[J]. Spectroscopy and Spectral Analysis, 2020, 40(12): 3772-3777.
12	吴铮, 张悦, 董泽. 基于改进高斯混合模型的热工过程异常值检测[J]. 系统仿真学报, 2023, 35(5): 1020-1033.
	Wu Zheng, Zhang Yue, Dong Ze. Outlier detection during thermal processes based on improved Gaussian mixture model[J]. Journal of System Simulation, 2023, 35(5): 1020-1033.
13	谢桦, 陈俊星, 赵宇明, 等. 基于SMOTE和决策树算法的电力变压器状态评估知识获取方法[J]. 电力自动化设备, 2020, 40(2): 137-142.
	Xie Hua, Chen Jun-xing, Zhao Yu-ming, et al. Knowledge acquisition method of power transformer condition assessment based on SMOTE and decision tree algorithm[J]. Electric Power Automation Equipment, 2020, 40(2): 137-142.
14	蔡瑞初, 白一鸣, 乔杰, 等. 基于混淆因子隐压缩表示模型的因果推断方法[J]. 计算机应用, 2021, 41(10): 2793-2798.
	Cai Rui-chu, Bai Yi-ming, Qiao Jie, et al. Causal inference method based on confounder hidden compact representation model[J]. Journal of Computer Applications, 2021, 41(10): 2793-2798.
15	张清华, 庞国弘, 李新太, 等. 基于代价敏感的序贯三支决策最优粒度选择方法[J]. 电子与信息学报, 2021, 43(10): 3001-3009.
	Zhang Qing-hua, Pang Guo-hong, Li Xin-tai, et al. Optimal granularity selection method based on cost-sensitive sequential three-way decisions[J]. Journal of Electronics & Information Technology, 2021, 43(10): 3001-3009.

相关文章 15

[1]	傅丽芳,陈卓,敖长林. 基于分类和回归树决策树的网络大数据集离群点动态检测算法[J]. 吉林大学学报(工学版), 2023, 53(9): 2620-2625.
[2]	马壮林,崔姗姗,胡大伟,王晋. 限行政策下传统小汽车出行者出行方式选择[J]. 吉林大学学报(工学版), 2023, 53(7): 1981-1993.
[3]	龙恩深,班光泽. 基于小波包信包提取的空调制冷压缩机怠速噪声诊断算法[J]. 吉林大学学报(工学版), 2023, 53(7): 1929-1934.
[4]	白琳,刘林军,李轩昂,吴沙,刘汝庆. 基于自监督学习的单目图像深度估计算法[J]. 吉林大学学报(工学版), 2023, 53(4): 1139-1145.
[5]	周怡娜,董宏丽,张勇,路敬祎. 基于VMD去噪和散布熵的管道信号特征提取方法[J]. 吉林大学学报(工学版), 2022, 52(4): 959-969.
[6]	尹超英,邵春福,黄兆国,王晓全,王晟由. 基于梯度提升决策树的多尺度建成环境对小汽车拥有的影响[J]. 吉林大学学报(工学版), 2022, 52(3): 572-577.
[7]	陈晓雷,孙永峰,李策,林冬梅. 基于卷积神经网络和双向长短期记忆的稳定抗噪声滚动轴承故障诊断[J]. 吉林大学学报(工学版), 2022, 52(2): 296-309.
[8]	李国发,王彦博,何佳龙,王继利. 机电装备健康状态评估研究进展及发展趋势[J]. 吉林大学学报(工学版), 2022, 52(2): 267-279.
[9]	杜先君,贾亮亮. 基于优化堆叠降噪自编码器的滚动轴承故障诊断[J]. 吉林大学学报(工学版), 2022, 52(12): 2827-2838.
[10]	吴静娴,申华鹏,韩印,杨敏. 考虑城市建成环境非线性作用的通勤时间模型[J]. 吉林大学学报(工学版), 2022, 52(11): 2568-2573.
[11]	戴礼灿,代翔,崔莹,魏永超. 基于深度集成学习的社交网络异常数据挖掘算法[J]. 吉林大学学报(工学版), 2022, 52(11): 2712-2717.
[12]	欧阳宁,李祖锋,林乐平. 基于多层次空⁃谱融合网络的高光谱图像分类[J]. 吉林大学学报(工学版), 2022, 52(10): 2438-2446.
[13]	黎才茂,陈少凡,林成蓉,候玉权,李浩. 基于循环知识图谱的虚拟社区知识动态推荐方法[J]. 吉林大学学报(工学版), 2022, 52(10): 2385-2390.
[14]	许鸿奎,姜彤彤,李鑫,姜斌祥,王永雷. 结合降噪自编码与极限学习机的LTE上行干扰分析[J]. 吉林大学学报(工学版), 2022, 52(1): 195-203.
[15]	刘桂霞,裴志尧,宋佳智. 基于深度学习的蛋白质⁃ATP结合位点预测[J]. 吉林大学学报(工学版), 2022, 52(1): 187-194.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed