Journal of Jilin University(Engineering and Technology Edition) ›› 2024, Vol. 54 ›› Issue (10): 2799-2806.doi: 10.13229/j.cnki.jdxbgxb.20230474

Previous Articles    

Optimization of infrared-visible road target detection by fusing GPNet and image multiscale features

Wen-cai SUN1(),Xu-ge HU1,Zhi-fa YANG1,2,3(),Fan-yu MENG2,Wei SUN3   

  1. 1.College of Transportation，Jilin University，Changchun 130022，China
    2.Product Planning and Project Management Department，China FAW Group Co. ，Ltd. ，Changchun 130013，China
    3.Changchun Automobile Industry Institute，Changchun 130031，China
  • Received:2023-05-12 Online:2024-10-01 Published:2024-11-22
  • Contact: Zhi-fa YANG E-mail:swcai@163.com;yangzf@jlu.edu.cn

RICH HTML

 2   

PDF (PC)

181

Abstract:

In order to improve the accuracy of road target detection in the field of road traffic safety， an innovative infrared and visible fusion and detection network is established by borrowing the idea of multi-level feature image fusion for fusion in image fusion technology and the idea of Ghost bottleneck module building in GPNet to reduce the complexity of the algorithm. The network is divided into three parts： selective image fusion module， lightweight target detection module and fusion quality and detection accuracy discriminative network. Three sets of experiments were conducted as data sets under urban working conditions with an average vehicle speed of 30-40 km/h in daytime， nighttime and special weather （rain， fog， etc.）. Experimental results： the highest average gradient lift of 5.648 81， cross-entropy of 0.936 68， edge strength of 56.945 7， information entropy of 0.925 208 781， mutual information of 1.000 548 571， peak signal-to-noise ratio 3.053 893 252， Qab0.342 882 208， Qcb0.208 983 81 and mean square error reduction of 0.08. The AP， mAP and Recall of the output of the lightweight target detection network are all at the optimal level， which verifies the advantages of the innovative application of infrared and visible light technologies for road obstacle detection.

Key words: engineering of communication and transportation, computer vision, infrared and visible image fusion, YOLOv5 target detection

CLC Number: 

  • U492.8

Fig.1

Network sctructure"

Fig.2

Selective image fusion pattern"

Fig.3

Feature extraction network structure"

Fig.4

Experimental route"

Fig.5

Infrared camera and visible light camera comparison chart"

Fig.6

Histogram equalized RGB image compared with the original image and distribution histogram"

Fig.7

Comparison of dark channel before and after defogging"

Table 1

Selective image pre-processing results"

方 法图像总数/张选择频次PSNR均值SSIM均值MSE均值综合指标均值
直方图均衡化350177390.898 60.03539.863 6
暗通道先验350173410.879 30.04441.835 3

Fig.8

Comparison of experimental images of typical image fusion strategies"

Fig.9

Innovate fusion framework fusion experiment"

Fig.10

Comparison of effect of fusion methods"

Table 2

Comparison of average time spent on various fusion image strategies"

方法用时/（s·帧-1）方法用时/（s·帧-1）
ADF4.56FPDE9.41
CBF68.11GFCE10.56
CNN140.51GTF16.16
HMSD_GF7.96LatLRR1 036.78
Hybrid_MSD36.46MGFF3.64
IFEVIP1.48MSVD6.78
本文2.65

Table 3

YOLOv5 detection effect"

方法

融合

车AP/%人AP/%平均AP/%Recall/%mAP50/%
IR75.484.379.8544.3249.7
VI70.180.775.4045.3146.5
CNN77.288.382.7546.2360.7
ADF77.582.379.9047.2151.7
LatLRR77.378.477.8547.8960.9
Hybrid_MSD67.378.773.0045.9861.7
IFEVIP77.179.178.145.2157.6
MGFF67.582.675.0544.9958.7
MSVD67.481.074.2047.786
TIF67.784.576.146.5264.3
本文83.288.986.0549.3465.7
1 纪嘉树.基于多传感器融合的无人驾驶环境感知技术研究[D].济南: 山东大学能源与动力工程学院,2022.
Ji Jia-Shu. Research on environment perception technology of unmanned driving based on multi-sensor fusion[D]. Ji'nan: School of Energy and Power Engineering,Shandong University, 2022.
2 张博雅. 红外与可见光数字图像融合技术研究[D].南京: 南京理工大学电子工程与光电技术学院, 2015.
Zhang Bo-Ya. Infrared and visible light digital image fusion technology research[D]. Nanjing: School of Electronic Engineering and Optoelectronic Technology, Nanjing University of Science & Technology, 2015.
3 刘畅. 基于深度卷积网络的多模图像融合与质量提升方法研究[D].衡阳: 南华大学电气工程学院, 2022.
Liu Chang. Research on multimodal image fusion and quality improvement method based on deep convolutional network[D]. Hengyang: School of Electrical Engineering,University of South China, 2022.
4 张婉婉. 基于深度学习的遥感图像融合算法研 究[D]. 烟台: 山东工商学院计算机科学与技术学院, 2022.
Zhang Wan-wan. Research on remote sensing image fusion algorithm on deep learning[D]. Yantai: College of Computer Science and Technology,Shandong Technology and Business University, 2022.
5 Fang L, Wang X, Wan Y. Adaptable active contour model with applications to infrared ship target segmentation[J].Journal of Electronic Imaging,2016,25(4):1093-1100.
6 Bavirisetti D P, Dhuli R, Fusion of infrared and visible sensor images based on anisotropic diffusion and karhunen-loeve transform[J]. IEEE Sensors Journal, 2016, 16(1): 203-209.
7 Kumar B K S. Image fusion based on pixel significance using cross bilateral filter[J]. Signal, Image and Video Processing, 2013, 9(5): 1193-1204.
8 Liu Y, Chen X, Cheng J, et al. Infrared and visible image fusion with convolutional neural networks[J]. International Journal of Wavelets, Multiresolution and Information Processing, 2018, 16(3): No.1850018.
9 高雪琴, 刘刚, 肖刚, 等. 基于FPDE的红外和可见光图像融合算法[J]. 自动化学报, 2020, 46(4): 796-804.
Gao Xue-Qin, Liu Gang, Xiao Gang, et al. Fusion algorithm of infrared and visible images based on FPDE[J]. Acta Automatica Sinica, 2020, 46(4): 796-804.
10 Zhou Z, Dong M, Xie X, et al. Fusion of infrared and visible images for night-vision context enhancement[J]. Appl Opt, 2016, 55(23): 6480-6490.
11 Zhang Y, Zhang L, Bai X Z, et al. Infrared and visual image fusion through infrared feature extraction and visual information preservation[J]. Infrared Physics & Technology,2017. 83: 227-237.
12 Bavirisetti D P, Xiao G, Zhao J, et al. Multi-scale guided image and video fusion[J]. Circuits, Systems, and Signal Processing, 2019, 38: 5576-5605.
13 Vasiliki B, Loukas B, Stefanos K, et al. Generating graph-inspired descriptors by merging ground-level and satellite data for robot localization[J]. Cybernetics and Systems, 2022, 13(1): 697-715.
14 Bavirisetti D P, Dhuli R. Two-scale image fusion of visible and infrared images using saliency detection[J]. Infrared Physics & Technology, 2016, 76: 52-64.
15 Redmon J, Divvala S, Girshick R, et al. You only look once: unified, real-time object detection[J/OL].[2015-01-10]. .
16 高明华,杨璨.基于改进卷积神经网络的交通目标检测方法[J].吉林大学学报: 工学版,2022,52(6):1353-1361.
Gao Ming-hua, Yang Can.Traffic target detection method based on improved convolution neural network[J].Journal of Jilin University(Engineering and Technology Edition), 2022, 52(6): 1353-1361.
[1] Xi-zhen ZHOU,He GONG,Dun-dun LI,Yan-jie JI,Jie YAN. Nonlinear model for impact of built environment on curb parking spaces occupancy [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(9): 2520-2530.
[2] Sheng-jie ZHU,Xuan WANG,Fang XU,Jia-qi PENG,Yuan-chao WANG. Multi-scale normalized detection method for airborne wide-area remote sensing images [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(8): 2329-2337.
[3] Ming-hui SUN,Hao XUE,Yu-bo JIN,Wei-dong QU,Gui-he QIN. Video saliency prediction with collective spatio-temporal attention [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(6): 1767-1776.
[4] Yu WANG,Kai ZHAO. Postprocessing of human pose heatmap based on sub⁃pixel location [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(5): 1385-1392.
[5] Dian-wei WANG,Chi ZHANG,Jie FANG,Zhi-jie XU. UAV target tracking algorithm based on high resolution siamese network [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(5): 1426-1434.
[6] Ya-qin QIN,Zheng-fu QIAN,Ji-ming XIE. Vehicle cooperative obstacle avoidance strategy driven by CLAM model and trajectory data [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(5): 1311-1322.
[7] Yun-long GAO,Ming REN,Chuan WU,Wen GAO. An improved anchor-free model based on attention mechanism for ship detection [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(5): 1407-1416.
[8] Ming-chen GU,Hui-yuan XIONG,Zeng-jun LIU,Qing-yu LUO,Hong LIU. Weight estimation model for trucks integrating multi-head attention mechanism [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(10): 2771-2780.
[9] Hong-tao LI,Lin-hong WANG,Jun-da LI. Influence of lighting and speed limit on visual search ability at highway intersections [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(8): 2287-2297.
[10] Wei-tiao WU,Kun ZENG,Wei ZHOU,Peng LI,Wen-zhou JIN. Deep learning method for bus passenger flow prediction based on multi-source data and surrogate-based optimization [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(7): 2001-2015.
[11] Zhen-liang LIU,Cun-bao ZHAO,Yun-peng WU,Mi-na MA,Long-shuang MA. Life⁃cycle seismic resilience assessment of highway bridge networks using data⁃driven method [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(6): 1695-1701.
[12] Hong-fei JIA,Ying-jun XU,Li-li YANG,Nan WANG. League member selection and benefit distribution of commercial vehicles multi⁃modal transportation [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(4): 1060-1069.
[13] Yu-lin CHANG,Wen-qian XU,Chao SUN,Peng ZHANG. Day⁃to⁃day equilibrium of hybrid traffic considering obedience degree under internet of vehicles environment [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(4): 1085-1093.
[14] Chao SUN,Hao-wei YIN,Wen-yun TANG,Zhao-ming CHU. Sensor deployment strategy and expansion inference of mobile phone data for traffic demand estimation [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(4): 1070-1077.
[15] Hui-zhen ZHANG,Zheng-kai GAO,Jian-qiang LI,Chen-xi WANG,Yu-biao PAN,Cheng WANG,Jing WANG. Short⁃term passenger flow forecasting of urban rail transit based on recurrent neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(2): 430-438.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright © Journal of Jilin University(Engineering and Technology Edition), All Rights Reserved.
Tel: +86-431-85095297　Fax: +86-431-85094074　E-mail: xbgxb@jlu.edu.cn
Powered by Beijing Magtech Co. Ltd