基于双孪生网络的可见光热红外目标实时鲁棒跟踪

doi:10.13229/j.cnki.jdxbgxb20210414

Abstract

Abstract:

Aiming at the problem of poor robustness in visible light and thermal infrared image fusion tracking using Siamese network architecture， a real-time RGB-Thermal （RGB-T） infrared target tracking method based on feature fusion of dual Siamese network was proposed. Firstly， dual Siamese network was used to extract the features of template branch and the search branch of the visible light and infrared images， and two different modalities feature layers were obtained. Secondly， the self-attention feature enhancement module （SFEM） was used to enhance the features of the two modalities， and the dual-modal feature fusion （DMFF） module was used to fuse the features of template branch and search branch respectively. Finally， the template branch and search branch were used for cross-correlation operation， and the target position was obtained by classification and regression branches， so as to complete target tracking.

The test results on the grayscale thermal infrared target tracking dataset (GTOT) show that the precision rate (PR) of the proposed method is 91.8%, the success rate (SR) is 78.1%, and the running speed is 60 $f / s$ . It shows that, compared with other RGB-T fusion tracking methods, the proposed method has higher robustness while maintaining real-time processing speed.

Key words: computer application, RGB-Thermal infrared, dual Siamese network, channel attention, spatial attention

CLC Number:

TP391.41

Lei FU,Wen-bin GU,Yong-bao AI,Wei LI,Nan ZHENG,Liu-yang WANG. Real⁃time robust RGB⁃T target tracking based on dual Siamese network[J].Journal of Jilin University(Engineering and Technology Edition), 2022, 52(12): 2906-2915.

Figures/Tables 10

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Table 2

Fig.6

Table 3

Fig.7

References 23

1	Lu Z, Rathod V, Votel R, et al. Retinatrack: online single stage joint detection and tracking[C]∥Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2020: 14656-14666.
2	徐涛, 马克, 刘才华. 基于深度学习的行人多目标跟踪方法[J]. 吉林大学学报:工学版, 2021, 51(1): 27-38.
	Xu Tao, Ma Ke, Liu Cai-hua. Multi object pedestrian tracking based on deep learning[J]. Journal of Jilin University (Engineering and Technology Edition), 2021, 51(1): 27-38.
3	孟琭, 杨旭. 目标跟踪算法综述[J]. 自动化学报, 2019, 45(7): 1244-1260.
	Meng Lu, Yang Xu. A survey of object tracking algorithms[J]. Acta Automatica Sinica, 2019, 45(7): 1244-1260.
4	苑晶, 李阳, 董星亮, 等. 基于运动模式在线分类的移动机器人目标跟踪[J]. 仪器仪表学报, 2017, 38(3): 568-577.
	Yuan Jing, Li Yang, Dong Xing-liang, et al. Target tracking with a mobile robot based on online classification for motion patterns[J]. Chinese Journal of Scientific Instrument, 2017, 38(3): 568-577.
5	Hao J, Zhou Y, Zhang G, et al. A review of target tracking algorithm based on UAV[C]∥Proceedings of the 2018 IEEE International Conference on Cyborg and Bionic Systems, Shenzhen, China, 2018: 328-333.
6	Zhang X, Ye P, Leung H, et al. Object fusion tracking based on visible and infrared images: a comprehensive review[J]. Information Fusion, 2020, 63: 166-187.
7	Li C, Wu X, Zhao N, et al. Fusing two-stream convolutional neural networks for RGB-T object tracking[J]. Neurocomputing, 2017, 281: 78-85.
8	Zhu Y, Li C, Luo B, et al. Dense feature aggregation and pruning for RGBT tracking[C]∥The 27th ACM International Conference, Nice, France, 2019: 465-472.
9	Gao Y, Li C, Zhu Y, et al. Deep adaptive fusion network for high performance RGBT tracking[C]∥2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Soule, South Korea, 2019: 91-99.
10	Li C, Lu A, Zheng A, et al. Multi-adapter RGBT tracking[C]∥2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, South Korea, 2019: 2262-2270.
11	Zhang X C, Ye P, Peng S Y, et al. SiamFT: an RGB-infrared fusion tracking method via fully convolutional Siamese networks[J]. IEEE Access, 2019, 7(1): 122122-122133.
12	申亚丽. 基于特征融合的RGBT双模态孪生跟踪网络[J]. 红外与激光工程, 2021, 50(3): 20200459.
	Shen Ya-li. RGBT dual-modal Siamese tracking network with feature fusion[J]. Infrared and Laser Engineering, 2021, 50(3): 20200459.
13	Xu Y, Wang Z, Li Z, et al. Towards robust and accurate visual tracking with target estimation guidelines[C]∥Association for the Advance of Artificial Intelligence (AAAI), New York, USA, 2020: 125491-125565.
14	Li B, Yan J, Wu W, et al. High performance visual tracking with Siamese region proposal network[C]∥Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, USA, 2017: 8971-8980.
15	Krizhevsky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional neural networks[C]∥Conference on Neural Information Processing Systems (NIPS), Lake Tahoe, USA, 2012: 1106-1164.
16	Fu J, Liu J, Tian H, et al. Dual attention network for scene segmentation[C]∥Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, USA, 2019: 3146-3154.
17	François C. Xception: deep learning with depthwise separable convolutions[C]∥Conference on Computer Vision and Pattern Recognition (CVPR), Hawai, USA, 2017: 1800-1807.
18	Lin T, Goyal P, Girshick R, et al. Focal loss for dense object detection[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2020, 42(2): 318-327.
19	Yu J, Jiang Y, Wang Z, et al. UnitBox: an advanced object detection network[C]∥ACM international conference on Multimedia, Amsterdam, Netherlands. 2016: 516-520.
20	Li C, Hui C, Hu S, et al. Learning collaborative sparse representation for grayscale-thermal tracking[J]. IEEE Transactions on Image Processing, 2016, 25(12): 5743-5756.
21	Li C, Liang X, Lu Y, et al. RGB-T object tracking: benchmark and baseline[J]. Pattern Recognition, 2019: 96: No. 106977.
22	Jung I, Son J, Beak M, et al. Real-time MDNet[C]∥European Conference on Computer Vision (ECCV), Munich, Germany, 2018: 89-104.
23	Zhang Z, Peng H. Deeper and wider Siamese networks for real-time visual tracking[C]∥Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, USA, 2019: 4591-4600.

Related Articles 15

[1]	Xian-yu QI,Wei WANG,Lin WANG,Yu-fei ZHAO,Yan-peng DONG. Semantic topological map building with object semantic grid map [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(2): 569-575.
[2]	Xiao-hu SHI,Jia-qi WU,Chun-guo WU,Shi CHENG,Xiao-hui WENG,Zhi-yong CHANG. Residual network based curve enhanced lane detection method [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(2): 584-592.
[3]	Peng GUO,Wen-chao ZHAO,Kun LEI. Dual⁃resource constrained flexible job shop optimal scheduling based on an improved Jaya algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(2): 480-487.
[4]	Jin-Zhen Liu,Guo-Hui Gao,Hui Xiong. Multi⁃scale attention network for brain tissue segmentation [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(2): 576-583.
[5]	Gui-he QIN,Jun-feng HUANG,Ming-hui SUN. Text input based on two⁃handed keyboard in virtual environment [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(8): 1881-1888.
[6]	Fu-heng QU,Tian-yu DING,Yang LU,Yong YANG,Ya-ting HU. Fast image codeword search algorithm based on neighborhood similarity [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(8): 1865-1871.
[7]	Tian BAI,Ming-wei XU,Si-ming LIU,Ji-an ZHANG,Zhe WANG. Dispute focus identification of pleading text based on deep neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(8): 1872-1880.
[8]	Feng-feng ZHOU,Hai-yang ZHU. SEE： sense EEG⁃based emotion algorithm via three⁃step feature selection strategy [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(8): 1834-1841.
[9]	Feng-feng ZHOU,Yi-chi ZHANG. Unsupervised feature engineering algorithm BioSAE based on sparse autoencoder [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(7): 1645-1656.
[10]	Jun WANG,Yan-hui XU,Li LI. Data fusion privacy protection method with low energy consumption and integrity verification [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(7): 1657-1665.
[11]	Yao-long KANG,Li-lu FENG,Jing-an ZHANG,Fu CHEN. Outlier mining algorithm for high dimensional categorical data streams based on spectral clustering [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(6): 1422-1427.
[12]	Wen-jun WANG,Yin-feng YU. Automatic completion algorithm for missing links in nowledge graph considering data sparsity [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(6): 1428-1433.
[13]	Xue-yun CHEN,Xue-yu BEI,Qu YAO,Xin JIN. Pedestrian segmentation and detection in multi-scene based on G-UNet [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(4): 925-933.
[14]	Shi-min FANG. Multiple source data selective integration algorithm based on frequent pattern tree [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(4): 885-890.
[15]	Da-xiang LI,Meng-si CHEN,Ying LIU. Spontaneous micro-expression recognition based on STA-LSTM [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(4): 897-909.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 10

[1]	Hu Zong-Jie，Wu Zhi-jun，Gao Guang-hai，Li Li-guang . Optimization of premixed diesel homogeneous charge preparation by spray hot-impingement[J]. 吉林大学学报(工学版), 2007, 37(01): 79 -84 .
[2]	Wang Qing-nian,Zheng Jun-feng，Wang Wei-hua . New adaptive control strategy of parallel hybrid electric bus[J]. 吉林大学学报(工学版), 2008, 38(02): 249 -0253 .
[3]	Dong Wei, Yu Xiu-min, Zhang You-kun . CVT control strategies for engine brake on long downhill of vehicle [J]. 吉林大学学报(工学版), 2006, 36(05): 650 -0653 .
[4]	. [J]. 吉林大学学报(工学版), 2007, 37(06): 1392 -1396 .
[5]	Dong Jing, Zhao Xiao-hui, Ying Na. Pitch detection algorithm based on dyadic wavelet transforms[J]. 吉林大学学报(工学版), 2006, 36(06): 978 -0982 .
[6]	HE Ren, CHEN Qiang-Zhang. Vehicle regenerative braking using dual switched reluctance motors/generators[J]. 吉林大学学报(工学版), 2009, 39(05): 1137 -1141 .
[7]	LI Suo-jun,GAO Hai-bo,DENG Zong-quan. Multi-objective optimization of rocker-bogie suspension parameters of lunar rover for energy-saving passing obstacle[J]. 吉林大学学报(工学版), 2010, 40(03): 729 -0734 .
[8]	Yang Zhong-zhen, Cui Cong . Modeling pollutant concentrations with artificial neural network technique[J]. 吉林大学学报(工学版), 2007, 37(03): 705 -0708 .
[9]	JIANG Hui, GUO Kong-Hui, ZHANG Jian-Wei. Design of automatic parallel parking steering controller based on pathplanning[J]. 吉林大学学报(工学版), 2011, 41(02): 293 -0297 .
[10]	MENG Ling-qi，DU Yong,MA Sheng-biao，GUO Bin . Nonlinearity of vertical vibration of medium plate mill [J]. 吉林大学学报(工学版), 2009, 39(03): 712 -0715 .

算法	OCC	LI	LSV	FM	TC	SO	DEF
SiamDW^［23］+RGB-T	67.5	70	68.9	71.1	63.5	76.4	69.1
RT-MDNet^［22］	73.3	77.2	79.1	78.1	73.7	85.6	73.1
DAPNet^［8］	87.3	90.0	84.7	82.3	89.3	93.7	91.9
DAFNet^［9］	87.3	89.9	82.2	80.9	89.8	93.9	94.7
MANet^［10］	88.2	91.4	86.9	84.7	88.9	93.2	92.3
SEFF	88.8	91.4	94.1	87.9	91.7	90.6	89.3

算法	OCC	LI	LSV	FM	TC	SO	DEF
SiamDW^［23］+RGB-T	53.6	58.8	56.5	57.6	51.7	58.5	58.2
RT-MDNet^［22］	57.6	63.8	63.7	64.1	59.0	63.4	61.0
DAPNet^［8］	67.4	72.2	64.8	61.9	69.0	69.2	77.1
DAFNet^［9］	68.4	72.7	66.5	64.2	70.3	69.8	76.5
MANet^［10］	69.6	73.6	70.6	69.4	70.2	70.0	75.2
SEFF	75.2	77.4	79.9	73.0	77.5	75.1	75.3

算法	显卡型号	速度/（f·s^-1）
DAPNet^［8］	1080Ti	2
DAFNet^［9］	1080Ti	26
MANet^［10］	1080	1.11
SEFF	1080Ti	60

Real⁃time robust RGB⁃T target tracking based on dual Siamese network

RICH HTML

PDF (PC)