基于随机性特征的加密和压缩流量分类

doi:10.13229/j.cnki.jdxbgxb20200314

Abstract

Abstract:

When encryption or compression algorithms are used to transmit data over the network, the payload data is generally random. Using existing traffic detection methods, it is difficult to effectively distinguish encrypted traffic from compressed traffic. To solve this problem, based on the differences between randomness of encrypted data and compressed data, this paper proposes the ECF randomness feature set. Without relying on the information of the network protocols, the packet headers, and the compression identifiers, the current mainstream machine learning algorithms are used to achieve accurate identification of encrypted or compressed data. Experiment results show that this method has higher accuracy compared with current methods and it also has good performance with generalization and migration.

Key words: traffic classification, encrypted traffic, compressed traffic, machine learning

CLC Number:

TP309.7

Guang-song LI,Wen-qing LI,Qing LI. Encrypted and compressed traffic classification based on random feature set[J].Journal of Jilin University(Engineering and Technology Edition), 2021, 51(4): 1375-1386.

Figures/Tables 15

Table 1

Table 2

Table 3

Fig.1

Fig.2

Table 4

Different encrypted and compressed data features"

编号	特征名称	简介	类型
ECF?1	卡方	检测目标数据与均匀分布数据之间的偏离程度，以字节的均匀分布作为理论推断值，取值按照式（1）给出的方法计算得出。	float
ECF?2	α?Renyi熵	检测目标数据的字节分布密度的随机性，以目标数据的实际字节分布作为离散的密度分布，取值根据式（2）给出的方法计算得出。	float
ECF?3	单比特频数	检验目标数据中0和1的比例与随机数据的相似性，取值为NIST测试集中Frequency Test检测值p_value。	float
ECF?4	块内频数	将目标数据分成若干子块，检测每一个子块中0和1的比例与随机数据的相似性，取值为NIST测试集中Frequency Test within a Block检测值p_value。	float
ECF?5	游程	检验目标数据中0或1的游程长度以及交替频率，确定数据中游程震荡是否过快或过慢，取值为NIST测试集中Runs Test检测值p_value。	float
ECF?6	最大游程	将目标数据分为若干子块，检测子块中1的最长游程与随机性数据的差距，取值为NIST测试集中Longest Run Of Ones检测值p_value。	float
ECF?7	傅里叶变换	检测目标数据的离散傅里叶变换的峰值高度，测试数据周期性与随机数据之间的偏差，取值为NIST测试集中Discrete Fourier Transform （Spectral） Test检测值p_value。	float
ECF?8	非重叠匹配	检测目标数据中固定字符串出现的频率，检测中一旦匹配及跳过匹配数据重新检索，取值为NIST测试集中Non?overlapping Template Matching Test检测值p_value。	float
ECF?9	序列化	检测目标数据中不同长度固定字符串出现的频率与随机性数据的差距，取值为NIST测试集中Serial Test检测值p_value1。	float
ECF?10	序列化	检测目标数据中不同长度固定字符串出现的频率与随机性数据的差距，取值为NIST测试集中Serial Test检测值p_value2。	float
ECF?11	累加和	检验目标数据在不同长度上的累加和与随机序列期望值的偏差，将目标数据中（0，1）转化为（-1，1）后正向计算累加和随机游走的最大偏移值，取值为NIST测试集中Cumulative Sums Test检测值p_value1。	float
ECF?12	累加和	检验目标数据在不同长度上的累加和与随机序列期望值的偏差，将目标数据中（0，1）转化为（-1，1）后正向计算累加和随机游走的最大偏移值，取值为NIST测试集中Cumulative Sums Test检测值p_value2。	float

Table 4

Fig.3

Table 5

Fig.4

Table 6

Fig.5

Fig.6

Fig.7

Table 7

Fig.8

References 19

1	王勇, 周慧怡, 俸皓,等. 基于深度卷积神经网络的网络流量分类方法[J]. 通信学报,2018, 39(1):14-23.
	Wang Yong, Zhou Hui-yi, Feng Hao, et al. Network traffic classification method basing on CNN[J]. Journal on Communications, 2018, 39(1):14-23.
2	Sandvine. 2018 Global Internet Phenomena Report[R]. Ontario Canda: Sandvine Incorporated ULC Waterloo, 2018.
3	潘吴斌, 程光, 郭晓军, 等. 网络加密流量识别研究综述及展望[J]. 通信学报, 2016, 37(9):154-167.
	Pan Wu-bin, Cheng Guang, Guo Xiao-jun, et al. Review and perspective on encrypted traffic identification research[J]. Journal on Communications, 2016, 37(9):154-167.
4	Dorfinger P, Panholzer G, John W. Entropy estimation for real-time encrypted traffic identification (short paper)[C]∥International Workshop on Traffic Monitoring and Analysis, Berlin,Germany,2011: 164-171.
5	朱玉娜, 韩继红, 袁霖, 等. 基于熵估计的安全协议密文域识别方法[J]. 电子与信息学报, 2016, 38(8): 1865-1871.
	Zhu Yu-na, Han Ji-hong, Yuan Lin, et al. Protocol Ciphertext Field Identification by Entropy Estimating[J]. Journal of Electronics & Information Technology, 2016, 38(8): 1865-1871.
6	赵博, 郭虹, 刘勤让, 等. 基于加权累积和检验的加密流量盲识别算法[J]. 软件学报, 2013, 24(6): 1334-1345.
	Zhao Bo, Guo Hong, Liu Qin-rang, et al. Protocol independent identification of encrypted traffic based on weighted eumnlative sum test[J]. Journal of Software, 2013, 24(6): 1334-1345.
7	King T, D'Agostino R B, Stephens M A. Goodness-of-fit techniques[J]. Journal of Educational Statistics, 1987, 12(4):412-416.
8	Shannon C E. Communication Theory of Secrecy Systems[J]. The Bell System Technical Journal, 1949, 28(4):656-715.
9	Malhotra P. Detection of encrypted streams for egress monitoring[D]. Malhotra, Paras:Iowa State University, 2007.
10	Conte T M, Wolfe A. Techniques for detecting encrypted data[P]. US:8799671,2014-08-05.
11	Hahn D, Apthorpe N, Feamster N. Detecting compressed cleartext traffic from consumer internet of things devices[J]. arXiv preprint arXiv:, 2018.
12	Casino F, Choo K K R, Patsakis C. HEDGE: efficient traffic classification of encrypted and compressed packets[J]. IEEE Transactions on Information Forensics and Security, 2019, 14(11): 2916-2926.
13	Wang R, Shoshitaishvili Y, Kruegel C, et al. Steal this movie: automatically bypassing DRM protection in streaming media services[C]∥ Proceedings of the 22nd USENIX conference on Security, Berkeley,USA,2013: 687-702.
14	Wang Y, Zhang Z, Guo L, et al. Using entropy to classify traffic more deeply[C]∥IEEE Sixth International Conference on Networking, Architecture, and Storage, Dalian, China, 2011: 45-52.
15	Khakpour A R, Liu A X. An information-theoretical approach to high-speed flow nature identification[J]. IEEE/ACM Transactions on Networking (TON), 2013, 21(4): 1076-1089.
16	雷博, 范九伦. 一维Renyi熵阈值法中参数的自适应选取[J]. 光子学报, 2009, 38(9):2439-2443.
	Lei Bo, Fan Jiu-lun. Self-adaptation preferences in one-dimensional Renyi entropy thresholding[J]. Acta Photonica Sinica, 2009, 38(9): 2439-2443.
17	Rukhin A L, Soto J, Nechvatal J R, et al. SP 800-22 Rev. 1a. A statistical test suite for random and pseudorandom number generators for cryptographic applications[S]. National Institute of Standards and Technology,USA,2010-09-16.
18	周志华. 机器学习[M]. 北京:清华大学出版社, 2016.
19	石竑松, 张翀斌, 杨永生,等. 随机性检测及其片面性[J]. 清华大学学报:自然科学版, 2011, 51(10):1269-1273.
	Shi Hong-song, Zhang Chong-bin,Yang Yong-sheng, et al. On randomness test and its incompleteness[J]. Journal of Tsinghua University(Science and Technology), 2011,51(10):1269-1273.

Related Articles 11

[1]	Xiao-long ZHU,Zhong XIE. Geospatial data extraction algorithm based on machine learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(3): 1011-1016.
[2]	Yang LI,Shuo LI,Li-wei JING. Estimate model based on Bayesian model and machine learning algorithms applicated in financial risk assessment [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(5): 1862-1869.
[3]	Wei FANG,Yi HUANG,Xin-qiang MA. Automatic defect detection for virtual network perceptual data based on machine learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(5): 1844-1849.
[4]	Zhou-zhou LIU,Wen-xiao YIN,Qian-yun ZHANG,Han PENG. Sensor cloud intrusion detection based on discrete optimization algorithm and machine learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(2): 692-702.
[5]	ZHAO Dong, ZANG Xue-bai, ZHAO Hong-wei. Random forest prediction method based on optimization of fruit fly [J]. 吉林大学学报(工学版), 2017, 47(2): 609-614.
[6]	XIA Jing-bo, BAI Jun, ZHAO Xiao-huan, WU Ji-xiang. Online network traffic classification using relevant vector machine [J]. 吉林大学学报(工学版), 2014, 44(2): 459-464.
[7]	WU Qi, LIU Jian-nan, KOU Wen-long, ZHANG Zong-sheng. Internet traffic identification by using improved one class support vector machines [J]. 吉林大学学报(工学版), 2013, 43(增刊1): 124-127.
[8]	TU Wei-wei, LI Ming, ZHOU Zhi-hua. Mining software defect factor [J]. 吉林大学学报(工学版), 2012, 42(增刊1): 382-386.
[9]	LIU Yuan-ning, SHEN Ting-jie, ZHANG Hao, LI Xin, WEI Qing-kai, HE Yu-zhe. New feature extraction methods of microRNA target genes [J]. 吉林大学学报(工学版), 2012, 42(02): 418-422.
[10]	GUO Kong-hui, WANG Xian-yun. Nonparametric models of shock absorber based on support vector machine regression [J]. 吉林大学学报(工学版), 2011, 41(增刊1): 1-4.
[11]	NI Ping, LIAO Jian-xin, ZHU Xiao-min,. Method for service level agreement measurement without negotiation [J]. 吉林大学学报(工学版), 2011, 41(01): 264-0269.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

类别	文件类型	来源	数量	大小
文本	txt	Multi?Domain Sentiment Dataset²	25	1.12 GB
图片	JPEG	SUN2012³	5000	935 MB
视频	avi	UCF101⁴	2500	852 MB
二进制文件	exe dll lib elf	win10、ubuntu 15系统文件	1910	1.33 GB
音频	MP3	网易音乐	198	1.03 GB
混合文档	PDF	互联网下载	472	814 MB

文件类型	原始大小	使用算法	最终大小	分割总数/份
文本	1.12 GB	AES、3DES、IDEA	1.12 GB	951 480
		RAR	221 MB	227 169
		ZIP	360 MB	368 784
		GZIP	362 MB	371 014
混合文件	814 MB	AES、3DES、IDEA	814 MB	833 267
		RAR	703 MB	719 866
		ZIP	734 MB	749 290
		GZIP	733 MB	750 763
二进制文件	1.33 GB	AES、3DES、IDEA	1.33 GB	1 403 012
		RAR	443 MB	453 626
		ZIP	566 MB	577 740
		GZIP	564MB	578 680
图片	935 MB	AES、3DES、IDEA	935 MB	954 959
		RAR	910 MB	930 238
		ZIP	911 MB	931 171
		GZIP	914 MB	933 469
视频	852 M	AES、3DES、IDEA	852 M	872 607
		RAR	840 M	859 172
		ZIP	839 M	858 454
		GZIP	841 M	860 094
音频	1.03 G	AES、3DES、IDEA	1.03 G	1 087 750
		RAR	1.01 G	1 068 870
		ZIP	1.02 G	1 070 249
		GZIP	1.02 G	1 074 898

模型	模型参数	D_1kB	D_2kB	D_4kB	D_8kB	D_16kB	D_32kB	D_64kB	detect
随机森林	n_estimators	400	200	200	200	200	200	200	305
	max_depth	12	12	12	12	12	12	12	12
	min_samples_split	100	100	100	100	100	100	100	2
	min_samples_leaf	50	50	50	50	50	50	50	1
	测试精度/%	71.42	77.9	86.58	92.02	97.07	98.92	99.51	73.9
Xgboost	n_estimators	600	600	600	500	400	400	400	400
	max_depth	12	12	12	12	12	12	12	12
	min_child_weight	400	400	400	400	400	400	200	3
	测试精度/%	71.46	78.06	86.67	92.06	97.04	98.76	99.45	73.23
MLP	hidden_layer_sizes	32，16	32，16	32，16	32，16	32，16	32，16	32，16	32，16
	max_iter	500	500	500	500	500	500	500	500
	测试精度/%	71.43	77.86	86.58	91.92	97.01	98.7	99.31	72.75

Encrypted and compressed traffic classification based on random feature set

RICH HTML

PDF (PC)