基于深度学习的交通事故救援图像融合算法

doi:10.13229/j.cnki.jdxbgxb.20230346

Abstract

Abstract:

Aiming at the problem that fire， smoke and other situations at the scene of serious traffic accidents affect the detection equipment to complete the search and rescue of trapped persons， a Convolutional Block Attention Module-Improved Loss-Dual-Discriminator Conditional Generative Based on Convolutional Attention Mechanism Adversarial Network （CBAM-IL-DDCGAN） infrared and visible image fusion method is proposed. Firstly， the decoding network with attention feature fusion module is used to restore and reconstruct the image from space and channel. Secondly， an adaptive weight calculation method based on gradient information is designed. Finally， the test experiment of fused image continuous frames was carried out. Experimental results show that the proposed image fusion algorithm performs well， and achieves a significant improvement of more than 7% in PSNR， SSIM and MSE compared with the traditional algorithm and the generative adversarial network algorithm. These results verify the feasibility and superiority of the fusion algorithm in complex traffic accident rescue.

Key words: transportation systems engineering, infrared image, visible light image, image fusion, discriminator, attention mechanism

CLC Number:

TP301.6

Sheng JIANG,Peng-lang WANG,Zhi-ji DENG,Yi-ming BIE. Image fusion algorithm for traffic accident rescue based on deep learning[J].Journal of Jilin University(Engineering and Technology Edition), 2023, 53(12): 3472-3480.

Figures/Tables 15

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Table 1

Table 2

Table 3

Table 4

Fig.11

References 19

1	李庆庆. 基于多层信息提取和传递的红外与可见光图像融合研究[D]. 长春:中国科学院长春光学精密机械与物理研究所, 2022.
	Li Qing-qing. Research on infrared and visible image fusion based on multilayer information extraction and transmission[D]. Changchun: Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, 2022.
2	程博阳. 基于滚动引导剪切波变换的红外与可见光图像融合研究[D]. 长春: 中国科学院长春光学精密机械与物理研究所, 2020.
	Cheng Bo-yang. Research on infrared and visible image fusion based on rolling guided shear wave transform[D]. Changchun: Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, 2020.
3	顾振飞, 袁小燕, 张登银, 等. 一种基于区域显著性识别的红外图像增强方法[J]. 江苏大学学报: 自然科学版, 2019, 40(6): 681-687.
	Gu Zhen-fei, Yuan Xiao-yan, Zhang Deng-yin, et al. A method of infrared image enhancement based on regional saliency recognition[J]. Journal of Jiangsu University (Natural Science Edition), 2019, 40(6): 681-687.
4	圣文顺, 孙艳文, 张会影. 基于稀疏理论与FFST-GIF的多聚焦图像融合算法[J]. 江苏大学学报: 自然科学版, 2022, 43(2): 195-200.
	Sheng Wen-shun, Sun Yan-wen, Zhang Hui-ying. Multi-focus image fusion algorithm based on sparseness theory and FFST-GIF[J]. Journal of Jiangsu University (Natural Science Edition), 2022, 43(2): 195-200.
5	王彬, 赵作鹏. 基于双流全卷积网络的驾驶员姿态估计方法[J]. 江苏大学学报: 自然科学版, 2022, 43(2): 161-168.
	Wang Bin, Zhao Zuo-peng. Driver attitude estimation method based on dual-stream fully convolutional network[J]. Journal of Jiangsu University(Natural Science Edition), 2022, 43(2): 161-168.
6	李臣旭, 江浩斌, 王成雨, 等. 基于传感器信息融合的车辆位姿估算方法[J]. 江苏大学学报: 自然科学版, 2022, 43(6): 636-644.
	Li Chen-xu, Jiang Hao-bin, Wang Cheng-yu, et al. Vehicle pose estimation method based on sensor information fusion[J]. Journal of Jiangsu University(Natural Science Edition), 2022, 43(6): 636-644.
7	Liu Y, Chen X, Peng H, et al. Multi-focus image fusion with a deep convolutional neural network[J]. Information Fusion, 2017, 36:191-207.
8	Zhao Y, Fu G, Wang H, et al. The fusion of unmatched infrared and visible images based on generative adversarial networks[J]. Mathematical Problems in Engineering, 2020, 2020: No. 3739040.
9	An W B, Wang H M. Infrared and visible image fusion with supervised convolutional Neural network[J]. Optik, 2020, 219: No. 165120.
10	Ma Jia-yi, Xu Han, Jiang Jun-jun, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995.
11	Fu Y, Wu X J, Durrani T. Image fusion based on generative adversarial network consistent with perception[J]. Information Fusion, 2021, 72: 110-125.
12	于鹏, 朴燕. 基于多尺度特征的行人重识别属性提取新方法[J]. 吉林大学学报: 工学版: 2023, 53(4): 1155-1162.
	Yu Peng, Pu Yan. New method for extracting pedestrian re-identification attributes based on multi-scale features[J]. Journal of Jilin University(Engineering and Technology Edition): 2023, 53(4): 1155-1162.
13	刘晶红, 邓安平, 陈琪琪, 等. 基于多重注意力机制的无锚框目标跟踪算法[J/OL]. [2022-12-01].
14	曾春艳, 严康, 王志锋, 等. 多尺度生成对抗网络下图像压缩感知重建[J/OL]. [2022-12-26].
15	申铉京, 张雪峰, 王玉, 等. 像素级卷积神经网络多聚焦图像融合算法[J]. 吉林大学学报: 工学版, 2022, 52(08):1857-1864.
	Shen Xuan-jing, Zhang Xue-feng, Wang Yu, et al. Pixel-level convolutional neural network multi-focus image fusion algorithm[J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(8): 1857-1864.
16	秦浩, 熊凌, 陈琳. 基于CBAM-DDcGAN的锌渣红外与可见光图像融合[J]. 武汉科技大学学报, 2023, 46(3): 216-224.
	Qin Hao, Xiong Ling, Chen Lin. Fusion of zinc slag infrared and visible light images based on CBAM-DDcGAN[J]. Journal of Wuhan University of Science and Technology, 2023, 46(3): 216-224.
17	黄倩. 基于深度学习的GUI源码自动生成[D]. 武汉: 武汉大学计算机学院, 2019.
	Hang Qian. GUI source code based on deep learning is automatically generated[D]. Wuhan: College of Computer Science and Technology, Wuhan University, 2019.
18	苏宁, 叶晗鸣, 王逍遥. 基于Adam优化的多传感器对接位姿融合算法研究[J]. 现代制造技术与装备,2022,58(2): 35-37, 41.
	Su Ning, Ye Han-ming, Wang Xiao-yao. Research on multi-sensor docking pose fusion algorithm based on Adam optimization[J]. Modern Manufacturing Technology and Equipment, 2022, 58(2): 35-37, 41.
19	张威. 生成式双波段彩色融合图像质量评价[J]. 电子设计工程, 2023, 31(5): 1-6.
	Zhang Wei. Generative dual-band color fusion image quality evaluation[J]. Electronic Design Engineering, 2023, 31(5): 1-6.

Related Articles 15

[1]	Guang HUO,Da-wei LIN,Yuan-ning LIU,Xiao-dong ZHU,Meng YUAN,Di GAI. Lightweight iris segmentation model based on multiscale feature and attention mechanism [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(9): 2591-2600.
[2]	Ming-yao XIAO,Xiong-fei LI,Rui ZHU. Medical image fusion based on pixel correlation analysis in NSST domain [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(9): 2640-2648.
[3]	Xiao-xin GUO,Jia-hui LI,Bao-liang ZHANG. Joint segmentation of optic cup and disc based on high resolution network [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(8): 2350-2357.
[4]	Fei-fei TANG,Hai-lian ZHOU,Tian-jun TANG,Hong-zhou ZHU,Yong WEN. Multi⁃step prediction method of landslide displacement based on fusion dynamic and static variables [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(6): 1833-1841.
[5]	Yan-tao TIAN,Xing HUANG,Hui-qiu LU,Kai-ge WANG,Fu-qiang XU. Multi⁃mode behavior trajectory prediction of surrounding vehicle based on attention and depth interaction [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(5): 1474-1480.
[6]	Wei LYU,Jia-ze HAN,Jing-hui CHU,Pei-guang JING. Multi⁃modal self⁃attention network for video memorability prediction [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(4): 1211-1219.
[7]	Yan-tao TIAN,Fu-qiang XU,Kai-ge WANG,Zi-xu HAO. Expected trajectory prediction of vehicle considering surrounding vehicle information [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(3): 674-681.
[8]	Zhou-zhou LIU,Chuan-xin SUN,Xiao-zhu WANG,Yang-mei ZHANG. Infrared and low⁃light image fusion based on VGG19 and low⁃pass filtering [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(1): 255-262.
[9]	Xuan-jing SHEN,Xue-feng ZHANG,Yu WANG,Yu-bo JIN. Multi⁃focus image fusion algorithm based on pixel⁃level convolutional neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(8): 1857-1864.
[10]	Ji-hong OUYANG,Ze-qi GUO,Si-guang LIU. Dual⁃branch hybrid attention decision net for diabetic retinopathy classification [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 648-656.
[11]	Xian-tong LI,Wei QUAN,Hua WANG,Peng-cheng SUN,Peng-jin AN,Yong-xing MAN. Route travel time prediction on deep learning model through spatiotemporal features [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 557-563.
[12]	Xiao⁃lei CHEN,Yong⁃feng SUN,Ce LI,Dong⁃mei LIN. Stable anti⁃noise fault diagnosis of rolling bearing based on CNN⁃BiLSTM [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(2): 296-309.
[13]	Da-ke ZHOU,Chao ZHANG,Xin YANG. Self-supervised 3D face reconstruction based on multi-scale feature fusion and dual attention mechanism [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(10): 2428-2437.
[14]	Xiong-fei LI,Jia-jing WU,Xiao-li ZHANG,Ze-yu WANG,Yun-cong FENG. Remote sensing image fusion algorithm based on relative total variation structure extraction [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(5): 1775-1784.
[15]	De-xing WANG,Ruo-you WU,Hong-chun YUAN,Peng GONG,Yue WANG. Underwater image restoration based on multi-scale attention fusion and convolutional neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(4): 1396-1404.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

指标	拉普拉斯算法	对抗网络算法	本文算法
PSNR	15.34	17.77	19.14
SSIM	0.73	0.78	0.86
MSE	1902.12	1087.34	792.65

序号	原融合图和可见图	本文算法融合图和可见图	提升幅度/%	原融合图和红外图	本文算法融合图和红外图	提升幅度/%
1	0.3976	0.5253	32.10	0.4927	0.5539	12.42
2	0.3958	0.5251	32.68	0.4932	0.5551	12.56
3	0.3939	0.5243	33.11	0.4934	0.5562	12.73
4	0.3959	0.5248	32.55	0.4942	0.5574	12.79
5	0.3980	0.5283	32.76	0.4941	0.5580	12.93

序号	原融合图和可见图	本文算法融合图和可见图	提升幅度/%	原融合图与红外图	本文算法融合图和红外图	提升幅度/%
1	12.2605	13.5765	10.73	13.849	14.843	7.17
2	12.2912	13.6010	10.66	13.860	14.855	7.18
3	12.2781	13.5963	10.74	13.843	14.847	7.25
4	12.2970	13.6041	10.63	13.872	14.865	7.16
5	12.3358	13.6566	10.71	13.880	14.862	7.07

序号	原融合图和可见图	本文算法融合图和可见图	提升幅度/%	原融合图和红外图	本文算法融合图和红外图	提升幅度/%
1	3863.9	2853.9	26.1	2680.1	2132.2	20.4
2	3836.7	2837.8	26.0	2673.6	2126.1	20.5
3	3848.3	2840.9	26.2	2683.9	2130.3	20.6
4	3831.6	2835.7	26.0	2666.1	2121.1	20.4
5	3797.5	2801.7	26.2	2661.1	2122.6	20.2

Image fusion algorithm for traffic accident rescue based on deep learning

RICH HTML

PDF (PC)