基于DNST和卷积稀疏表示的红外与可见光图像融合

doi:10.13229/j.cnki.jdxbgxb20200166

摘要/Abstract

摘要：

为了克服传统红外与可见光图像融合方法的不足，本文提出了一种基于离散不可分离剪切波与卷积稀疏表示的融合方法。首先，利用离散不可分离剪切波将源图像分解为近似图像和方向细节图像，相比较于其他多尺度分解工具，离散不可分离剪切波能够在不同尺度内更好地分离出图像中重叠的重要特征信息。其次，利用源图像的显著特征图加权平均融合近似图像，保持融合图像的亮度和能量不丢失。卷积稀疏表示能够深度提取图像的显著特征，利用多维系数的l₁范数作为活性测度构造显著特征图，生成近似图像的权重分配决策图。方向细节图像的融合规则采用“系数绝对值取大-高斯滤波”规则，通过“系数绝对值取大”规则获得初始权重分配决策图，利用高斯滤波器对决策图进行滤波处理，降低噪声的敏感度，同时增加可见光图像信息比例。最后，通过离散不可分离剪切波逆变换对融合后的系数进行重构，得到最后的融合图像。实验结果表明，相比较于已有文献的其他典型融合方法，本文融合方法在主观视觉和客观评价准则方面都取得了较好的融合性能。

关键词: 计算机应用, 图像融合, 离散不可分离剪切波, 卷积稀疏表示, 显著特征图

Abstract:

In order to overcome the shortcomings of traditional infrared and visible image fusion methods，a fusion method based on Discrete Nonseparable Shearlet Transform （DNST） and Convolution Sparse Representation （CSR） is proposed. Firstly， the source images are decomposed into approximate images and directional detail images using DNST. Compared with other multi-scale decomposition tools， DNST can better separate the overlapped important feature information in different scales. Secondly， the salient feature maps of the source images are employed to weight average approximate images， which can prevent the loss of the brightness and energy. CSR can deeply extract the salient features of the image， The l1 norm of multi-dimensional coefficients is used as the activity level measure to construct the Salient Feature Map （SFM）， which can generate the weight distribution decision map of the approximate image. The rule of Coefficient absolute max-Gaussian filtering is used as fusion rule of the directional detail images. The decision map of initial weight distribution is obtained by the Coefficient absolute max rule， then the decision map is filtered by Gaussian filter to reduce the sensitivity of noise and increase the proportions of visible image information. Finally， the fused coefficients are reconstructed by the inverse DNST， and the fusion image is obtained. Experimental results demonstrate that the proposed fusion method can achieve superior performance compared with other typical fusion methods in the existing literature in both subjective vision and objective criteria evaluation.

Key words: computer application, image fusion, discrete nonseparable shearlet transform, convolutional sparse representation, salient feature map

中图分类号:

TP391.41

陈广秋,陈昱存,李佳悦,刘广文. 基于DNST和卷积稀疏表示的红外与可见光图像融合[J]. 吉林大学学报(工学版), 2021, 51(3): 996-1010.

Guang-qiu CHEN,Yu-cun CHEN,Jia-yue LI,Guang-wen LIU. Infrared and visible image fusion based on discrete nonseparable shearlet transform and convolutional sparse representation[J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(3): 996-1010.

图/表 17

图1

图2

图3

图4

图5

表1

图6

图7

表2

图8

表3

图9

表4

图10

表5

图11

表6

参考文献 40

1	Lee M W, Kwak K C. Performance comparison of infrared and visible image fusion approaches[C]∥International Conference on Control, Artificial Intelligence, Robotics & Optimization (ICCAIRO), Prague, Czech Republic,2017:274-277.
2	Ma J, Ma Y, Li C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019,45:153-178.
3	刘斌,辛迦楠, 谌文江,等. 不可分拉普拉斯金字塔构造及其在多光谱图像融合中的应用[J]. 计算机应用, 2019,39(2):564-570.
	Liu Bin, Xin Jia-nan, Chen Wen-jiang, et al. Construction of non-separable Laplacian pyramid and its application in multi-spectral image fusion[J]. Journal of Computer Applications, 2019, 39(2):564-570.
4	李雄飞,宋璐,张小利. 基于协同经验小波变换的遥感图像融合[J]. 吉林大学学报:工学版,2019,49(4):1307-1319.
	Li Xiong-fei, Song Lu, Zhang Xiao-li. Remote sensing image fusion based on cooperative empirical wavelet transform [J]. Journal of Jilin University (Engineering and Technology Edition),2019,49(4):1307-1319.
5	郭全民,王言,李翰山. 改进IHS-Curvelet变换融合可见光与红外图像抗晕光方法[J]. 红外与激光工程,2018,47(11):1126002.
	Guo Quan-min, Wang Yan, Li Han-shan. Anti-halation method of visible and infrared image fusion based on Improved IHS-Curvelet transform[J]. Infrared and Laser Engineering, 2018,47(11):1126002.
6	刘哲,徐涛,宋余庆,等. 基于NSCT变换和相似信息鲁棒主成分分析模型的图像融合技术[J]. 吉林大学学报:工学版, 2018,48(5):1614-1620.
	Liu Zhe, Xu Tao, Song Yu-qing, et al. Image fusion technology based on NSCT and robust principal component analysis model with similar information[J]. Journal of Jilin University (Engineering and Technology Edition), 2018,48(5):1614-1620.
7	Sun Z, Hu H. Off-line fusion of intravascular ultrasound and optical coherence tomography images[J]. Journal of Medical Imaging and Health Informatics,2017,7(7):1531-1538.
8	Li S, Kang X, Fang L, et al. Pixel-level image fusion: a survey of the state of the art[J]. Information Fusion, 2016, 33:100-112.
9	Kutyniok G, Lim W Q, Reisenhofer R. ShearLab 3D: faithful digital shearlet transforms based on compactly supported shearlets[J]. ACM Transactions on Mathematical Software, 2016,42(1):No.5.
10	邱泽敏. 结合区域与边缘特征的红外与可见光图像融合算法[J]. 红外技术, 2018,40(5):53-58.
	Qiu Ze-min. Infrared and visible image fusion algorithm combined with regional characteristics and edge characteristics[J]. Infrared Technology, 2018, 40(5):53-58.
11	Jin X, Jiang Q, Yao S, et al. A survey of infrared and visual image fusion methods[J]. Infrared Physics & Technology, 2017, 85:478-501.
12	Liu Y, Chen X, Wang Z, et al. Deep learning for pixel-level image fusion: Recent advances and future prospects[J]. Information Fusion, 2018, 42:158-173.
13	Zhang Qiang,Liu Yi, Blum R S, et al. Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: a review[J]. Information Fusion,2018,40:50-75.
14	Liu Y, Chen X, Ward R, et al. Image fusion with convolutional sparse representation[J]. IEEE Signal Processing Letters, 2016, 23(12):1882-1886.
15	Wohlberg B. Efficient algorithms for convolutional sparse representations[J]. IEEE Transactions on Image Processing, 2016, 25(1):301-315.
16	Xydeas C, Petrovic V. Objective image fusion performance measure[J]. Electronics Letters, 2000,36(4):308-309.
17	Ma K, Zeng K, Wang Z. Perceptual quality assessment for multi-exposure image fusion[J]. IEEE Transactions on Image Processing, 2015, 24(11):3345-3356.
18	Aslantas V, Bendes E. A new image quality metric for image fusion: The sum of the correlations of differences[J]. AEU-International Journal of Electronics and Communications, 2015, 69(12):1890-1896.
19	Chen Y, Blum R S. A new automated quality assessment algorithm for image fusion[J]. Image and Vision Computing,2009,27(10):1421-1432.
20	Haghighat M, Razian M A. Fast-FMI: non-reference image fusion metric[C]∥2014 IEEE 8th International Conference on Application of Information and Communication Technologies (AICT), Astana, Kazakhstan,2014:1-3.
21	Zhao W, Lu H. Medical image fusion and denoising with alternating sequential filter and adaptive fractional order total variation[J]. IEEE Transactions on Instrumentation and Measurement, 2017, 66(9):2283-2294.
22	Zhao J, Feng H, Xu Z, et al. Detail enhanced multi-source fusion using visual weight map extraction based on multi scale edge preserving decomposition[J]. Optics Communications, 2013, 287:45-52.
23	Zhu Jin, Jin Wei-qi, Li Li, et al. Fusion of the low-light-level visible and infrared images for night-vision context enhancement[J]. Chinese Optics Letters, 2018, 16(1):013501.
24	Du J, Li W, Xiao B. Anatomical-functional image fusion by information of interest in local laplacian filtering domain[J]. IEEE Transactions on Image Processing, 2017, 26(12):5855-5866.
25	Hu J, Li S. The multiscale directional bilateral filter and its application to multisensor image fusion[J]. Information Fusion, 2012,13(3):196-206.
26	da Cunha A L, Zhou J, Do M N. The nonsubsampled contourlet transform: theory, design, and applications[J]. IEEE Transactions on Image Processing, 2006, 15(10):3089-3101.
27	Li S, Yang B, Hu J. Performance comparison of different multi-resolution transforms for image fusion[J]. Information Fusion, 2011, 12(2):74-84.
28	Li H, Liu L, Huang W, et al. An improved fusion algorithm for infrared and visible images based on multi-scale transform[J]. Infrared Physics & Technology, 2016,74:28-37.
29	杨风暴, 赵艳霞, 吉琳娜,等. 基于局部能量匹配的红外偏振图像融合[J]. 红外技术, 2016(4):319-324.
	Yang Feng-bao, Zhao Yan-xia, Ji Lin-na, et al. An infrared polarization image fusion method based on local energy matching[J]. Infrared Technology, 2016(4): 319-324.
30	Qu Xiao-bo, Yan Jing-wen, Xiao Hong-zhi,et al. Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain[J]. Acta Automatica Sinica, 2008, 34(12):1508-1514.
31	Ma J, Zhou Z, Wang B, et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics & Technology, 2017, 82:8-17.
32	Jin Xin, Jiang Qian, Yao Shao-wen, et al. Infrared and visual image fusion method based on discrete cosine transform and local spatial frequency in discrete stationary wavelet transform domain[J]. Infrared Physics & Technology, 2018,88:1-12.
33	Liu Y, Liu S, Wang Z. A general framework for image fusion based on multi-scale transform and sparse representation [J]. Information Fusion, 2015, 24:147-164.
34	Liu C H, Qi Y, Ding W R. Infrared and visible image fusion method based on saliency detection in sparse domain[J]. Infrared Physics & Technology, 2017,83:94-102.
35	Liu Y, Wang Z. Simultaneous image fusion and denoising with adaptive sparse representation[J]. IET Image Processing, 2015, 9(5):347-357.
36	Zhan Kun, Xie Yuan-ge, Wang Hai-bo, et al. Fast filtering image fusion[J]. Journal of Electronic Imaging, 2017,26(6):063004.
37	Li Wen, Xie Yuan-ge Y.. Zhou Hao-le,et al. Structure-aware image fusion[J]. Optik, 2018, 172:1-11.
38	Ma J, Chen C, Li C, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31:100-109.
39	Zhou Z, Wang B, Li S, et al. Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with Gaussian and bilateral filters[J]. Information Fusion, 2016,30:15-26.
40	Li H, Wu X J, Durrani T S. Infrared and visible image fusion with ResNet and zero-phase component analysis[J]. Infrared Physics & Technology, 2019,102:103039.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

分解尺度数	方向数目
1	8
2	8 16
3	8 8 16
4	8 8 16 16
5	8 8 8 16 16
6	8 8 8 16 16 16

方法	Q^AB/F	SSIM	SCD	Q_HVS	FMI_w	时间/s
MEPFl₀	0.3171	0.5814	1.5541	0.5136	0.2775	9.375
MGF	0.3437	0.6141	1.5865	0.5338	0.3672	3.058
LLP	0.1311	0.3982	1.3853	0.4636	0.2915	1531.465
MDBF	0.4077	0.6570	1.6331	0.5646	0.3689	940.703
NSCT	0.4140	0.6588	1.6462	0.5917	0.3944	914.553
DNST	0.4203	0.6693	1.6517	0.5938	0.4072	23.767

融合规则	Q^AB/F	SSIM	SCD	Q_HVS	FMI_w	时间/s
PCA_GRD	0.4007	0.6053	1.2592	0.4196	0.3425	208.323
EM_VAR	0.4188	0.6305	1.3596	0.4072	0.3446	1501.886
SF_PCNN	0.3907	0.6167	1.2326	0.3507	0.3371	229.933
VSM_WLS	0.4073	0.6377	1.2884	0.3947	0.3509	32.549
DCT_LSF	0.4271	0.6319	1.3601	0.4089	0.3646	1939.591
CSR_GF	0.4308	0.6538	1.3736	0.4520	0.3702	563.768

图像	评价指标	DWT_SR	NSCT_SR	JSR	JSR_SD	ASR	DNST_CSRGF
Soldier_behind_smoke	Q^AB/F	0.4619	0.4532	0.4475	0.3791	0.4424	0.4836
	SSIM	0.6529	0.4999	0.6431	0.6245	0.6388	0.7102
	SCD	1.3343	1.3544	1.4619	1.4872	1.4253	1.5264
	Q_HVS	0.5125	0.4556	0.4951	0.5091	0.4099	0.5513
	FMI_w	0.3707	0.3879	0.2409	0.2211	0.4007	0.4186

图像	评价指标	FFIF	SAIF	GTF	HMSD	ResNet50	DNST_CSRGF
Kaptein_1123	Q^AB/F	0.4429	0.3525	0.3144	0.4295	0.3865	0.4901
	SSIM	0.6476	0.6456	0.6978	0.7258	0.7231	0.7439
	SCD	1.2461	1.2237	0.9689	1.6196	1.6076	1.6703
	Q_HVS	0.4331	0.4524	0.3329	0.4759	0.4652	0.4938
	FMI_w	0.3969	0.3805	0.4056	0.3514	0.3860	0.4152