基于多尺度级联卷积神经网络的高光谱图像分析

doi:10.13229/j.cnki.jdxbgxb.20220096

吉林大学学报(工学版) ›› 2023, Vol. 53 ›› Issue (12): 3547-3557.doi: 10.13229/j.cnki.jdxbgxb.20220096

• 计算机科学与技术 • 上一篇

基于多尺度级联卷积神经网络的高光谱图像分析

朱逢乐¹(),刘益^2,^3,⁴,乔欣¹,何梦竹^2,³,郑增威^2,³,孙霖^2,³()

^1.浙江工业大学机械工程学院，杭州 310023
^2.浙大城市学院计算机与计算科学学院，杭州 310015
^3.智能植物工厂浙江省工程实验室，杭州 310015
^4.浙江大学计算机科学与技术学院，杭州 310027

收稿日期:2022-01-25 出版日期:2023-12-01 发布日期:2024-01-12
通讯作者: 孙霖 E-mail:zhufl@zjut.edu.cn;sunl@zucc.edu.cn
作者简介:朱逢乐（1988-），女，讲师，博士.研究方向：农业信息智能感知，机器学习.E-mail：zhufl@zjut.edu.cn
基金资助:
浙江省自然科学基金项目(LGN22F020002);浙江省重点研发计划项目(2023C02010);国家自然科学基金项目(62072402)

Analysis of hyperspectral image based on multi-scale cascaded convolutional neural network

Feng-le ZHU¹(),Yi LIU^2,^3,⁴,Xin QIAO¹,Meng-zhu HE^2,³,Zeng-wei ZHENG^2,³,Lin SUN^2,³()

^1.College of Mechanical Engineering，Zhejiang University of Technology，Hangzhou 310023，China
^2.School of Computer & Computing Science，Hangzhou City University，Hangzhou 310015，China
^3.Intelligent Plant Factory of Zhejiang Province Engineering Laboratory，Hangzhou 310015，China
^4.College of Computer Science and Technology，Zhejiang University，Hangzhou 310027，China

Received:2022-01-25 Online:2023-12-01 Published:2024-01-12
Contact: Lin SUN E-mail:zhufl@zjut.edu.cn;sunl@zucc.edu.cn

摘要/Abstract

摘要：

在样本有限情况下对象级别的叶片高光谱图像建模中，提出了多尺度三维和一维级联卷积神经网络模型。首先，在三维卷积神经网络（3D-CNN）中嵌入扩张卷积增大卷积核感受野，构建了多尺度3D-CNN，提取和融合不同尺度的光谱-空间联合特征，在不增加网络参数的情况下提升了模型性能。然后，对最优多尺度3D-CNN网络级联一维卷积神经网络（1D-CNN），进一步降低计算复杂度和过拟合程度。最后，在罗勒叶片叶绿素含量回归和辣椒叶片干旱胁迫识别两类数据集上进行最优网络框架探究并对比了一系列基准CNN模型。结果表明，对于叶片高光谱图像回归和分类，本文模型均能在小样本条件下有效提升模型泛化性能并降低计算复杂度。

关键词: 农业电气化与自动化, 高光谱图像, 化学计量学, 多尺度级联卷积神经网络, 扩张卷积, 植物表型

Abstract:

In object-level hyperspectral image modeling with limited leaf samples， a multi-scale three dimensional-one dimensional cascaded convolution neural network model was proposed. Firstly， in 3D convolution neural network （3D-CNN）， the dilated convolution was embedded to increase the receptive field of convolution kernel， and a multi-scale 3D-CNN network was constructed to extract and fuse spectral-spatial joint features of different scales， to improve the model performance without increasing the number of network parameters. Then， the one-dimensional convolutional neural network （1D-CNN） was cascaded to the optimal multi-scale 3D-CNN network， to further reduce the computational complexity and overfitting degree of model. Finally， on two datasets of chlorophyll content regression for basil leaf and drought stress recognition for pepper leaf， the optimal network architecture was explored， with comparison to a series of baseline CNN models. Experimental results showed that the proposed model can effectively improve the generalization performance and reduce the computational complexity for both regression and classification tasks of leaf hyperspectral images under the condition of small samples.

Key words: agricultural electrification and automation, hyperspectral images, chemometrics, multi-scale cascaded convolutional neural network, dilated convolution, plant phenotyping

中图分类号:

朱逢乐,刘益,乔欣,何梦竹,郑增威,孙霖. 基于多尺度级联卷积神经网络的高光谱图像分析[J]. 吉林大学学报(工学版), 2023, 53(12): 3547-3557.

Feng-le ZHU,Yi LIU,Xin QIAO,Meng-zhu HE,Zeng-wei ZHENG,Lin SUN. Analysis of hyperspectral image based on multi-scale cascaded convolutional neural network[J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(12): 3547-3557.

图/表 13

表1

图1

图2

图3

图4

表2

表3

表4

表5

表6

图5

表7

图6

参考文献 26

1	Yu Ke-qiang, Zhao Yan-ru, Zhu Feng-le, et al. Mapping of chlorophyll and SPAD distribution in pepper leaves during leaf senescence using visible and near-infrared hyperspectral imaging[J]. Transactions of the ASABE, 2016, 59(1): 13-24.
2	Zhang Chu, Wu Wen-yan, Zhou Lei, et al. Developing deep learning based regression approaches for determination of chemical compositions in dry black goji berries (Lycium ruthenicum Murr.) using near-infrared hyperspectral imaging[J]. Food Chemistry, 2020, 319: No. 126536.
3	Hadji I, Wildes R P. What do we understand about convolutional networks?[J/OL]. [2022-01-02].
4	Zhu Su-su, Zhou Lei, Gao Pan, et al. Near-infrared hyperspectral imaging combined with deep learning to identify cotton seed varieties[J]. Molecules, 2019, 24: No. 3268.
5	Yang Wei, Yang Ce, Hao Zi-yuan, et al. Diagnosis of plant cold damage based on hyperspectral imaging and convolutional neural network[J]. IEEE Access, 2019, 7: 118239-118248.
6	Rehman T U, Ma D, Wang L, et al. Predictive spectral analysis using an end-to-end deep model from hyperspectral images for high-throughput plant phenotyping[J]. Computers and Electronics in Agriculture, 2020, 177: No. 105713.
7	He Wei-wen, He Hong-yuan, Wang Fang-lin, et al. Non-Destructive detection and recognition of pesticide residues on garlic chive (Allium Tuberosum) leaves based on short wave infrared hyperspectral imaging and one-dimensional convolutional neural network[J]. Journal of Food Measurement and Characterization, 2021, 15: 4497-4507.
8	Park K, Hong Y K, Kim G H, et al. Classification of apple leaf conditions in hyper-spectral images for diagnosis of Marssonina blotch using mRMR and deep neural network[J]. Computers and Electronics in Agriculture, 2018, 148: 179-187.
9	Zhang Jin-nuo, Yang Yong, Feng Xu-ping, et al. Identification of bacterial blight resistant rice seeds using terahertz imaging and hyperspectral imaging combined with convolutional neural network[J]. Frontiers in Plant Science, 2020, 11: No.821.
10	Polder G, Blok P M, de Villiers H A C, et al. Potato virus Y detection in seed potatoes using deep learning on hyperspectral images[J]. Frontiers in Plant Science, 2019, 10: No.209.
11	Pang Lei, Sen Men, Yan Lei, et al. Rapid vitality estimation and prediction of corn seeds based on spectra and images using deep learning and hyperspectral imaging techniques[J]. IEEE Access, 2020, 8: 123026-123036.
12	Nagasubramanian K, Jones S, Singh A K, et al. Plant disease identification using explainable 3D deep learning on hyperspectral images[J]. Plant Methods, 2019, 15: No. 98.
13	Yan Tian-ying, Xu Wei, Lin Jiao, et al. Combining multi-dimensional convolutional neural network (CNN) with visualization method for detection of Aphis gossypii glover infection in cotton leaves using hyperspectral imaging[J]. Frontiers in Plant Science, 2021, 12: No. 604510.
14	Kamilaris A, Prenafeta-Boldú F X. Deep learning in agriculture: a survey[J]. Computers and Electronics in Agriculture, 2018, 147: 70-90.
15	Hara K, Kataoka H, Satoh Y. Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet?[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018: 6546-6555.
16	Roy S K, Krishna G, Dubey S R, et al. HybridSN: exploring 3-D-2-D CNN feature hierarchy for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17(2): 277-281.
17	张祥东, 王腾军, 朱劭俊, 等. 基于扩张卷积注意力神经网络的高光谱图像分类[J]. 光学学报, 2021, 41(3): 43-53.
	Zhang Xiang-dong, Wang Teng-jun, Zhu Shao-jun, et al. Hyperspectral image classification based on dilated convolutional attention neural network[J]. Acta Optica Sinica, 2021, 41(3): 43-53.
18	李运丽. 光强对罗勒叶中花青素含量及光合特性的影响[D]. 南京: 南京农业大学园艺学院, 2011.
	Li Yun-li. Effects of light intensity on anthocyanins contents and photosynthetic characteristics in Ocimum basilicum L . leaves[D]. Nanjing: College of Horticulture, Nanjing Agricultural University, 2011.
19	张德炎. 水分胁迫对辣椒叶片生理活性的影响[J]. 湖北农业科学, 2010, 49(4): 819-821.
	Zhang De-yan. Effect of water stress on some physiology indices of pepper leaves[J]. Hubei Agricultural Sciences, 2010, 49(4): 819-821.
20	Tran D, Bourdev L, Fergus R, et al. Learning spatiotemporal features with 3D convolutional networks[C]∥Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 2015: 4489-4497.
21	Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions[J/OL]. [2022-01-05].
22	He Kai-ming, Zhang Xiang-yu, Ren Shao-qing, et al. Deep residual learning for image recognition[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016: 770-778.
23	Cruz A, Ampatzidis Y, Pierro R, et al. Detection of grapevine yellows symptoms in Vitis vinifera L. with artificial intelligence[J]. Computers and Electronics in Agriculture, 2019, 157: 63-76.
24	Ollinger S V. Sources of variability in canopy reflectance and the convergent properties of plants[J]. New Phytologist, 2011, 189: 375-394.
25	蔡倩, 白伟, 郑家明, 等. 水分胁迫对春玉米光合特性及水分利用效率的影响[J]. 辽宁农业科学, 2021, 5: 1-6.
	Cai Qian, Bai Wei, Zheng Jia-ming, et al. Effect of water stress on photosynthetic characteristics and water use efficiency of spring maize[J]. Liaoning Agricultural Sciences, 2021, 5: 1-6.
26	汤东, 程平, 杨建军, 等. 天山北坡山前植物对干旱胁迫的生理响应[J]. 干旱区研究, 2021, 38(6): 1683-1694.
	Tang Dong, Cheng Ping, Yang Jian-jun, et al. Physiological responses of plants to drought stress in the Northern Piedmont, Tianshan Mountains[J]. Arid Zone Research, 2021, 38(6): 1683-1694.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

样本集	样本数	最大值	最小值	均值	标准差
训练集	380	47.00	6.80	31.39	6.97
验证集	80	46.70	8.10	31.08	6.89
测试集	80	45.50	11.10	30.24	6.94

模型	训练方式	验证集		测试集
模型	训练方式	R²	RMSE	R²	RMSE
1D-CNN	手动提取特征	0.7529	3.3902	0.7432	3.4993
2D-CNN	端到端	0.8606	2.5969	0.8415	2.6789
3D-CNN	端到端	0.9433	1.5981	0.8726	2.4204

模型	训练方式	验证集		测试集
模型	训练方式	Accuracy	F₁-score	Accuracy	F₁-score
1D-CNN	手动提取特征	0.6231	0.6411	0.6001	0.6299
2D-CNN	端到端	0.6900	0.6988	0.6617	0.6794
3D-CNN	端到端	0.7792	0.8023	0.7111	0.7457

模型	嵌入的扩张3D卷积核	验证集		测试集
模型	嵌入的扩张3D卷积核	R²	RMSE	R²	RMSE
基准3D-CNN （3×3×3，d=1）		0.9433	1.5981	0.8726	2.4204
多尺度3D-CNN	3×3×3 （d=2）	0.9501	1.4005	0.8922	2.2675
	3×3×3 （d=3）	0.9498	1.4193	0.8868	2.3239
	3×3×3 （d=4）	0.9398	1.6821	0.8768	2.4139

模型	嵌入的扩张 3D卷积核	验证集		测试集
模型	嵌入的扩张 3D卷积核	Accuracy	F₁-score	Accuracy	F₁-score
基准3D-CNN （3×3×3，d=1）		0.7792	0.8023	0.7111	0.7457
多尺度 3D-CNN	3×3×3 （d=2）	0.8031	0.8145	0.7409	0.7522
	3×3×3 （d=3）	0.7900	0.8019	0.7316	0.7389
	3×3×3 （d=4）	0.7821	0.7892	0.7198	0.7272

基于多尺度级联卷积神经网络的高光谱图像分析

Analysis of hyperspectral image based on multi-scale cascaded convolutional neural network

RICH HTML

PDF (PC)

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 26

相关文章 7

Metrics

本文评价

推荐阅读 0

不同转换位点的多尺度3D-1D-CNN模型	训练时长（s·epoch^-1）	验证集		测试集
不同转换位点的多尺度3D-1D-CNN模型	训练时长（s·epoch^-1）	R²	RMSE	R²	RMSE
①（即基准 1D-CNN）	9	0.7529	3.3902	0.7432	3.4993
②	42	0.7783	3.2274	0.7699	3.2999
③	43	0.8014	3.0190	0.7935	3.1171
④	45	0.8188	2.9661	0.8156	2.9744
⑤	47	0.8722	2.4213	0.8563	2.7080
⑥	50	0.9045	2.1165	0.8849	2.3793
⑦	55	0.9316	1.8211	0.9101	2.0468
⑧	63	0.9385	1.7036	0.9031	2.1150
⑨	78	0.9502	1.4113	0.8998	2.1987
⑩（即最佳多尺度3D-CNN）	80	0.9501	1.4005	0.8922	2.2675

[1]	欧阳宁,李祖锋,林乐平. 基于多层次空⁃谱融合网络的高光谱图像分类[J]. 吉林大学学报(工学版), 2022, 52(10): 2438-2446.
[2]	王柯俨, 李云松, 宋娟, 廖惠琳, 吴宪云. 基于局部边缘预测的空谱联合高光谱图像无损压缩[J]. 吉林大学学报(工学版), 2017, 47(2): 677-685.
[3]	齐滨, 赵春晖, 王玉磊. 基于支持向量机与相关向量机的高光谱图像分类[J]. 吉林大学学报(工学版), 2013, 43(增刊1): 143-147.
[4]	王瀛, 郭雷, 梁楠. 基于核熵成分分析的高光谱遥感图像分类算法[J]. , 2012, (06): 1597-1601.
[5]	赵春晖,胡春梅. 基于目标正交子空间投影加权的高光谱图像异常检测算法[J]. 吉林大学学报(工学版), 2011, 41(05): 1468-1474.
[6]	付朝阳,郭雷,常威威. 网络的高光谱图像融合[J]. 吉林大学学报(工学版), 2011, 41(03): 838-843.
[7]	王立国，赵春晖，毕晓君 . 端元选择算法在波段选择中的应用[J]. 吉林大学学报(工学版), 2007, 37(04): 915-919.

不同转换位点的多尺度3D-1D-CNN模型	训练时长（s·epoch^-1）	验证集		测试集
不同转换位点的多尺度3D-1D-CNN模型	训练时长（s·epoch^-1）	Accuracy	F₁-score	Accuracy	F₁-score
①（即基准 1D-CNN）	10	0.6231	0.6411	0.6001	0.6299
②	39	0.6337	0.6344	0.6297	0.6123
③	42	0.6552	0.6781	0.6535	0.6588
④	43	0.6885	0.6957	0.6781	0.6973
⑤	47	0.7537	0.7423	0.7324	0.7232
⑥	51	0.7699	0.7771	0.7480	0.7569
⑦	56	0.7916	0.8002	0.7715	0.7791
⑧	63	0.7926	0.7855	0.7689	0.7542
⑨	81	0.7995	0.7912	0.7534	0.7628
⑩（即最佳多尺度3D-CNN）	82	0.8031	0.8145	0.7409	0.7522