基于多层次空⁃谱融合网络的高光谱图像分类

doi:10.13229/j.cnki.jdxbgxb20210298

Abstract

Abstract:

To acquire fine spectral and spatial features and their interaction information in hyperspectral image classification， a hierarchical spatial-spectral fusion network is proposed. Firstly， the hierarchical features extraction module was exploited to extract the spectral and spatial features of hyperspectral image respectively. Secondly， the spatial-spectral feature interactive fusion module was designed and employed to fuse the features and produce the joint spatial-spectral features. The proposed network can not only extract and integrate the fine spatial and spectral features of different levels，but also capture the interaction between spectral and spatial features by joint learning. The experimental results show that the proposed network performed better than the state-of-the-art neural network-based classification methods. The network is shown the capability of extracting fine features and capturing the spatial-spectral joint features for classification.

Key words: hyperspectral image classification, hierarchical features extraction module, spatial-spectral feature interactive fusion module, feature fusion

CLC Number:

TP753

Ning OUYANG,Zu-feng LI,Le-ping LIN. Hyperspectral image classification based on hierarchical spatial-spectral fusion network[J].Journal of Jilin University(Engineering and Technology Edition), 2022, 52(10): 2438-2446.

Figures/Tables 12

Fig.1

Fig.2

Table 1

Parameter setting of spatial-spectral hierarchical feature extraction module"

操作类型	HFEM_spa	HFEM_spe
步幅卷积size /strides	$2 × 2 × 1,12 2 × 2 × 1, s t r i d e s$	$1 × 1 × 2,12 1 × 1 × 2, s t r i d e s$
常规卷积size /strides	$2 × 2 × 1,12 1 × 1 × 1, s t r i d e s$	$1 × 1 × 2,12 1 × 1 × 1, s t r i d e s$
Conv size /strides	$2 × 2 × 2,512 1 × 1 × 1, s t r i d e s$	$1 × 1 × 2,512 1 × 1 × 1, s t r i d e s$
Reshape	$512 × S 1$	$512 × S 2$

Table 1

Fig.3

Table 2

Table 3

Table 4

Table 5

Fig.4

Fig.5

Fig.6

Table 6

References 20

1	Li W, Prasad S, Flower J E, et al. Locality-preserving dimensionality reduction and classification for hyperspectral image analysis[J]. IEEE Transactions on Geoscience & Remote Sensing,2012,50(4):1185-1198.
2	Li W, Chen C, Su H J, et al. Local binary patterns and extreme learning machine for hyperspectral imagery classification[J]. IEEE Transactions on Geoence & Remote Sensing, 2015, 53(7): 3681-3693.
3	Huang K, Li S, Kang X, et al. Spectral-spatial hyperspectral image classification based on KNN[J]. Sensing and Imaging, 2016, 17(1):1-13.
4	闫敬文, 陈宏达, 刘蕾. 高光谱图像分类的研究进展[J]. 光学精密工程, 2019, 27(3): 680-693.
	Yan Jing-wen, Chen Hong-da, Liu Lei. Overiew of hyperspectral image classification[J]. Optics and Precision Engineering, 2019, 27(3): 680-693.
5	Zhang H, Li Y, Zhang Y, et al. Spectral-spatial classification of hyperspectral imagery using a dual-channel convolutional neural network[J]. Remote Sensing Letters, 2017, 8(5):438-447.
6	Ouyang N, Zhu T, Lin L P. Convolutional neural network trained by joint loss for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2018, 16(3): 457-461.
7	Qing Y, Liu W. Hyperspectral image classification based on multi-scale residual network with attention mechanism[J]. Remote Sensing,2021,13(3):No.335.
8	Zhu M H, Jiao L C, Liu F, et al. Residual spectral–spatial attention network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(1): 449-462.
9	欧阳宁, 朱婷, 林乐平. 基于空-谱融合网络的高光谱图像分类方法[J].计算机应用, 2018, 38(7): 1888-1892.
	Ouyang Ning, Zhu Ting, Lin Le-ping. Spatial-spetral fusion network for hyperspectral image classification method[J]. Journal of Computer Applications, 2018, 38(7): 1888-1892.
10	Roy S K, Krishna G, Dubey S R, et al. HybridSN: exploring 3D-2D CNN feature hierarchy for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17(2): 277-281.
11	Li Y, Zhang H K, Shen Q. Spectral-spatial classification of hyperspectral imagery with 3D convolutional neural network[J]. Remote Sensing, 2017, 9(1): 67-87.
12	Zhang C, Li G, Du S, et al. Three-dimensional densely connected convolutional network for hyperspectral remote sensing image classification[J]. Journal of Applied Remote Sensing, 2019, 13(1):No.16519.
13	Zhang C J, Li G D, Du S H, et al. Multi-scale dense networks for hyperspectral remote sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(11): 9201-9222.
14	Wang W, Dou S, Jiang Z, et al. A fast dense spectral-spatial convolution network framework for hyperspectral images classification[J]. Remote Sensing, 2018, 10(7): No.1068.
15	Zhu M H, Jiao L C, Liu F, et al. Residual spectral–spatial attention network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(1): 449-462.
16	Huang G, Chen D, Li T, et al. Multi-scale dense networks for resource efficient image classification[C]//International Conference on Learning Representations, Vancouver, Canada, 2018: 1-14.
17	Chang Y H, Xu J T, Gao Z Y. Multi-scale dense attention network for stereo matching[J]. Electronics, 2020, 9(11):1881-1892.
18	Lu J, Yang J W, Batra D, et al. Hierarchical question-image co-attention for visual question answering[C]//Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain, 2016: 289-297.
19	Ma H Y, Li Y J, Ji X P, et al. MsCoa: multi-step co-attention model for multi-label classification[J]. IEEE Access, 2019, 7: 109635-109645.
20	Li M N, Tei K J, Fukazawa Y. An efficient adaptive attention neural network for social recommendation[J]. IEEE Access, 2020, 8: 63595-63606.

Related Articles 13

[1]	Xiao-ying PAN,De WEI,Yi-zhe ZHAO. Detecetion of lung nodule based on mask R-CNN and contextual convolutional neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(10): 2419-2427.
[2]	Da-ke ZHOU,Chao ZHANG,Xin YANG. Self-supervised 3D face reconstruction based on multi-scale feature fusion and dual attention mechanism [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(10): 2428-2437.
[3]	Hong-wei ZHAO,Dong-sheng HUO,Jie WANG,Xiao-ning LI. Image classification of insect pests based on saliency detection [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2174-2181.
[4]	WANG Sheng-sheng, GUO Xu, ZHANG Jia-chen, WANG Guang-yao, ZHAO Xin. Shape recognition algorithm based on fusion of global and local properties [J]. 吉林大学学报(工学版), 2016, 46(5): 1627-1632.
[5]	ZHANG Hao, LIU Hai-ming, WU Chun-guo, ZHANG Yan-mei, ZHAO Tian-ming, LI Shou-tao. Detection method of vehicle in highway green toll lane based on multi-feature fusion [J]. 吉林大学学报(工学版), 2016, 46(1): 271-276.
[6]	YANG Xin,LIU Jia,ZHOU Peng-yu,ZHOU Da-ke. Adaptive particle filter for object tracking based on fusing multiple features [J]. 吉林大学学报(工学版), 2015, 45(2): 533-539.
[7]	WU Di, CAO Jie. Multi-feature fusion face recognition based on KRWDA algorithm under smart environment [J]. 吉林大学学报(工学版), 2013, 43(02): 439-443.
[8]	WANG Xin-ying, LIU Gang, GU Fang-ming, XIAO Wei. Heterogeneous feature fusion method based on semantic and shape for 3D model retrieval [J]. 吉林大学学报(工学版), 2012, 42(增刊1): 359-363.
[9]	QU Zhi-guo, WANG Ping, GAO Ying-hui, WANG Peng, SHEN Zhen-kang, LI Jiang. Edge detection based on feature fusion of USAN area [J]. , 2012, (03): 759-765.
[10]	CHEN Mian-Shu, FU Ping, LI Yong, ZHANG Hui. Image feature fusion based on scope similarity scores minimization [J]. 吉林大学学报(工学版), 2010, 40(增刊): 365-0368.
[11]	WANG Ying, LI Wen-Hui. Highprecision video flame detection algorithm base on multifeature fusion [J]. 吉林大学学报(工学版), 2010, 40(03): 769-0775.
[12]	SHANG Fei, MA Jun-Xiao, YAO Li, TIAN Di, QIU Chun-Ling. Multifeature fusion based method for monitoring working status of instruments [J]. 吉林大学学报(工学版), 2010, 40(02): 545-0548.
[13]	ZHENG Ya-yu,TIAN Xiang,CHEN Yao-wu. Visual attention model based on fussion of spatiotemporal features [J]. 吉林大学学报(工学版), 2009, 39(06): 1625-1630.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

方法	3D-CNN	3D-DenseNet	MSDN（n=4）	HSFF_C	HSFF
1	97.23	97.31	99.10	97.29	100
2	98.95	96.51	96.73	98.32	99.59
3	96.87	93.88	99.85	99.25	99.49
4	95.77	94.95	94.88	97.84	98.90
5	90.68	98.97	94.13	99.64	99.55
6	92.72	97.72	98.95	99.55	99.96
7	94.68	92.19	98.20	99.52	99.47
8	99.82	98.21	95.81	99.69	100
9	84.63	93.01	99.70	98.33	100
10	98.01	96.20	93.80	99.13	99.08
11	98.81	97.41	96.05	99.60	99.70
12	95.41	95.03	96.94	92.67	99.59
13	99.35	99.86	95.74	99.79	100
14	99.19	99.19	99.39	99.16	99.83
15	93.09	95.52	95.50	97.43	98.90
16	96.08	93.43	95.88	89.19	96.12
OA/%	96.60±2.58	96.92±0.40	97.06±0.60	98.70±0.41	99.57±0.11
AA/%	95.71±1.61	96.21±1.24	96.87±1.13	97.90±0.35	99.38±0.17
Kappa×100	96.13±2.92	96.49±0.46	96.64±0.69	98.52±0.47	99.50±0.12

方法	3D?CNN	3D?DenseNet	MSDN（n=4）	HSFF_C	HSFF
1	98.63	99.65	99.87	99.96	99.81
2	98.49	91.26	99.48	99.33	99.65
3	92.05	91.37	91.98	93.55	98.90
4	89.65	92.53	90.96	95.51	95.95
5	62.34	91.06	87.78	94.14	93.36
6	96.89	93.42	95.94	98.64	99.32
7	97.56	95.05	93.04	97.56	99.49
8	97.31	96.74	95.86	98.81	99.77
9	99.81	95.24	94.66	99.97	100
10	99.44	96.74	99.40	100	100
11	100	99.47	99.96	100	99.86
12	99.62	98.15	99.08	99.31	100
13	100	99.91	100	100	100
OA/%	96.19±1.54	96.23±1.99	97.36±0.43	98.92±0.28	99.41±0.17
AA/%	94.75±1.57	95.43±1.73	96.00±0.58	98.21±0.50	98.93±0.39
Kappa×100	95.76±1.71	95.80±2.22	97.07±0.47	98.80±0.31	99.35±0.19

方法	3D?CNN	3D?DenseNet	MSDN（n=4）	HSFF_C	HSFF
1	93.33	98.42	98.33	99.66	99.87
2	99.22	99.88	99.94	99.94	99.96
3	96.15	96.79	97.59	99.28	99.63
4	99.84	99.78	99.88	99.90	99.91
5	100	99.38	99.93	99.98	100
6	94.15	99.76	99.91	99.56	99.98
7	99.93	97.59	99.12	99.87	99.98
8	87.85	97.17	96.76	98.45	99.35
9	99.93	99.62	99.96	100	99.95
OA/%	96.33±1.99	99.15±0.15	99.26±0.07	99.69±0.06	99.87±0.04
AA/%	96.67±2.03	98.71±0.25	99.04±0.14	99.62±0.09	99.84±0.04
Kappa×100	95.12±2.66	98.87±0.20	99.02±0.10	99.58±0.09	99.83±0.06

方法	3D?CNN	3D?DenseNet	MSDN（n=4）	HSFF_C	HSFF
1	99.97	98.83	99.81	99.91	99.99
2	99.24	100	99.80	100	100
3	99.86	99.91	99.79	99.77	100
4	99.00	99.71	99.70	99.84	99.88
5	99.71	99.95	99.83	99.64	99.99
6	99.98	99.97	99.76	99.89	99.99
7	71.35	99.99	100	100	100
8	99.94	99.45	99.66	99.89	99.92
9	98.40	99.99	99.78	99.43	100
10	99.73	99.63	99.51	99.55	99.94
11	99.71	99.30	99.66	99.72	99.98
12	99.97	99.98	100	99.67	99.96
13	99.54	99.62	100	99.79	99.93
14	96.37	99.75	98.03	99.51	100
15	91.09	98.42	99.82	99.80	99.82
16	97.75	100	99.48	100	100
OA/%	89.94±2.14	99.56±0.10	99.68±0.11	99.72±0.06	99.95±0.03
AA/%	96.98±0.69	99.66±0.13	99.66±0.12	99.70±0.10	99.96±0.02
Kappa×100	88.67±2.44	99.51±0.11	99.71±0.09	99.77±0.07	99.94±0.03

数据	t/s		3D?DenseNet	MSDN（n=4）	HSFF_C	HSFF
IP	训练	1309.3	6136.7	9106.2	2122.7	3898.3
IP	测试	5.5	26.8	40.7	11.8	19.8
UP	训练	1694.1	7394.6	6796.1	3879.1	5526.9
UP	测试	14.7	60.9	61.9	39.6	56.9
KSC	训练	619.2	3205.1	2008.5	1028.3	1809.6
KSC	测试	2.5	12.8	9.1	5.7	7.1
Epochs		300	200	200	300	300

Hyperspectral image classification based on hierarchical spatial-spectral fusion network

RICH HTML

PDF (PC)