像素级卷积神经网络多聚焦图像融合算法

doi:10.13229/j.cnki.jdxbgxb20211096

Abstract

Abstract:

In this paper， a novel convolutional neural network（CNN） for multi-focus image fusion is proposed. Compared with existing image fusion methods based on CNN which decompose the source image into several patches and adopt a classifier to estimate whether the patch is focused or defocused， the method in this paper directly converts the whole image into a decision diagram. The pixel-level regression strategy can make use of the complementary information and address the difficulty of estimating blur level around the focused/defocused region. Furthermore， the ringed residual network（RResNet） block is utilized to extract more semantic information from the focused region in image fusion field. In the meanwhile， the structural similarity index（SSIM） loss is utilized to estimate the structural similarity between the generated fusion image and the ground-truth reference to improve the quality of the fused images， and the edge preservation loss function is applied to preserve more gradient information from source image. Experimental results demonstrate that the proposed method is superior to other fusion algorithms in subjective visual effect and objective assessment.

Key words: multi-focus image fusion, deep learning, pixel-level regression, convolutional neural network

CLC Number:

TP391

Xuan-jing SHEN,Xue-feng ZHANG,Yu WANG,Yu-bo JIN. Multi⁃focus image fusion algorithm based on pixel⁃level convolutional neural network[J].Journal of Jilin University(Engineering and Technology Edition), 2022, 52(8): 1857-1864.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Fig.5

Table 2

Fig.6

Table 3

Fig.7

Table 4

References 21

1	Li S, Kang X, Fang L, et al. Pixel-level image fusion: a survey of the state of the art[J]. Information Fusion, 2017, 33: 100-112.
2	Huang W, Jing Z. Evaluation of focus measures in multi-focus image fusion[J]. Pattern Recognition Letters, 2007, 28(4): 493-500.
3	Aslantas V, Kurban R. Fusion of multi-focus images using differential evolution algorithm[J]. Expert Systems with Applications, 2010, 37(12): 8861-8870.
4	Liu Y, Liu S, Wang Z. Multi-focus image fusion with dense SIFT[J]. Information Fusion, 2015, 23: 139-155.
5	Hill P, Al-Mualla M E, Bull D. Perceptual image fusion using wavelets[J]. IEEE Transactions on Image Processing, 2016, 26(3): 1076-1088.
6	Yu B, Jia B, Ding L, et al. Hybrid dual-tree complex wavelet transform and support vector machine for digital multi-focus image fusion[J]. Neurocomputing, 2016, 182: 1-9.
7	张强,郭宝龙.基于Curvelet变换的图像融合算法[J]. 吉林大学学报: 工学版, 2007, 37(2): 458-463.
	Zhang Qiang, Guo Bao-long. Image fusion algorithm based on Curvelet transform[J]. Journal of Jilin University(Engineering and Technology Edition), 2007, 37(2): 458-463.
8	孙伟, 郭宝龙, 陈龙. 非降采样Contourlet域方向区域多聚焦图像融合算法[J]. 吉林大学学报: 工学版, 2009, 39(5): 1384-1389.
	Sun Wei, Guo Bao-long, Chen Long. Multi-focus image fusion algorithm for non-down-sampling Contourlet domain direction region[J]. Journal of Jilin University(Engineering and Technology Edition), 2009, 39(5): 1384-1389.
9	Bhatnagar G, Wu Q M J, Liu Z. Directive contrast based multimodal medical image fusion in NSCT domain[J]. IEEE Transactions on Multimedia, 2013, 15(5): 1014-1024.
10	Liu X, Mei W, Du H. Multi-modality medical image fusion based on image decomposition framework and nonsubsampled shearlet transform[J]. Biomedical Signal Processing and Control, 2018, 40: 343-350.
11	Liu Y, Chen X, Peng H, et al. Multi-focus image fusion with a deep convolutional neural network[J]. Information Fusion, 2017, 36: 191-207.
12	Zhang Y, Liu Y, Sun P, et al. IFCNN: a general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118.
13	Amin-Naji M, Aghagolzadeh A, Ezoji M. Ensemble of CNN for multi-focus image fusion[J]. Information fusion, 2019, 51: 201-214.
14	Gai D, Shen X, Chen H, et al. Multi-focus image fusion method based on two stage of convolutional neural network[J]. Signal Processing, 2020, 176: 107681.
15	Ma B, Zhu Y, Yin X, et al. Sesf-fuse: an unsupervised deep model for multi-focus image fusion[J]. Neural Computing and Applications, 2021, 33(11): 5793-5804.
16	Zhang H, Le Z, Shao Z, et al. MFF-GAN: an unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion[J]. Information Fusion, 2021, 66: 40-53.
17	Li S, Kang X, Hu J. Image fusion with guided filtering[J]. IEEE Transactions on Image Processing, 2013, 22(7): 2864-2875.
18	Fakhari F, Mosavi M R, Lajvardi M M. Image fusion based on multi-scale transform and sparse representation: an image energy approach[J]. Iet Image Processing, 2017, 11(11): 1041-1049.
19	Zhou Z, Li S, Wang B. Multi-scale weighted gradient-based fusion for multi-focus images[J]. Information Fusion, 2014, 20: 60-72.
20	Liu Y, Chen X, Ward R K, et al. Image fusion with convolutional sparse representation[J]. IEEE Signal Processing Letters, 2016, 23(12): 1882-1886.
21	Liu Y, Chen X, Peng H, et al. Multi-focus image fusion with a deep convolutional neural network[J]. Information Fusion, 2017, 36: 191-207.

Related Articles 15

[1]	Ming-hua GAO,Can YANG. Traffic target detection method based on improved convolution neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(6): 1353-1361.
[2]	Huai-jiang YANG,Er-shuai WANG,Yong-xin SUI,Feng YAN,Yue ZHOU. Simplified residual structure and fast deep residual networks [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(6): 1413-1421.
[3]	Xue-zhi WANG,Qing-liang LI,Wen-hui LI. Spatio⁃temporal model of soil moisture prediction integrated with transfer learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 675-683.
[4]	Xiang-jun LI,Jie-ying TU,Zhi-bin ZHAO. Validity classification of melting curve based on multi⁃scale fusion convolutional neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 633-639.
[5]	Ji-hong OUYANG,Ze-qi GUO,Si-guang LIU. Dual⁃branch hybrid attention decision net for diabetic retinopathy classification [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 648-656.
[6]	Xian-tong LI,Wei QUAN,Hua WANG,Peng-cheng SUN,Peng-jin AN,Yong-xing MAN. Route travel time prediction on deep learning model through spatiotemporal features [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 557-563.
[7]	Lin SONG,Li-ping WANG,Jun WU,Li-wen GUAN,Zhi-gui LIU. Reliability analysis based on cyber⁃physical system and digital twin [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(2): 439-449.
[8]	Jie CAO,Jia-lin MA,Dai-lin HUANG,Ping YU. A fault diagnosis method based on multi Markov transition field [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(2): 491-496.
[9]	Wen-zhi GAO,Yan-jun WANG,Xin-wei WANG,Pan ZHANG,Yong LI,Yang DONG. Real⁃time diagnosis for misfire fault of diesel engine based on convolutional neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(2): 417-424.
[10]	Long ZHANG,Tian-peng XU,Chao-bing WANG,Jian-yu YI,Can-zhuang ZHEN. Gearbox fault diagnosis baed on convolutional gated recurrent network [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(2): 368-376.
[11]	Liang DUAN,Chun-yuan SONG,Chao LIU,Wei WEI,Cheng-ji LYU. State recognition in bearing temperature of high-speed train based on machine learning algorithms [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(1): 53-62.
[12]	You QU,Wen-hui LI. Single-stage rotated object detection network based on anchor transformation [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(1): 162-173.
[13]	Gui-xia LIU,Zhi-yao PEI,Jia-zhi SONG. Prediction of protein-ATP binding site based on deep learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(1): 187-194.
[14]	Jie ZHANG,Wen JING,Fu CHEN. Vulnerability detection of instant messaging network protocol based on passive clustering algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2253-2258.
[15]	Hui ZHONG,Heng KANG,Ying-da LYU,Zhen-jian LI,Hong LI,Ruo-chuan OUYANG. Image manipulation localization algorithm based on channel attention convolutional neural networks [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(5): 1838-1844.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

方法	AG	EI	STD	VIF	Q^AB/F	MI
CNN^［21］	6.8744	71.3997	64.3632	0.9406	0.7145	5.5862
SESF^［15］	6.9354	72.0408	64.4824	0.9439	0.7144	5.6415
MWGF^［19］	6.7442	70.1093	64.1820	0.9244	0.6995	5.3950
MST_SR^［18］	6.8996	71.6012	64.2876	0.9437	0.7031	4.8962
DSIFT^［4］	6.9370	72.0509	64.4518	0.9427	0.7149	5.6584
ECNN^［13］	6.9173	71.8495	64.4488	0.9400	0.7134	5.7165
IFCNN^［12］	6.9339	71.9895	64.2728	0.9386	0.6890	4.6187
CSR^［20］	4.7882	50.9524	62.7171	0.7939	0.5026	4.8318
GF^［17］本文	6.7656 6.9459	70.1844 72.0919	64.0856 64.4815	0.9283 0.9444	0.7059 0.7153	5.0743 5.6633

方法	AG	EI	STD	VIF	Q^AB/F	MI
CNN^［21］	4.7918	50.4950	61.0154	0.8640	0.5169	4.1046
SESF^［15］	4.8493	51.0865	61.0706	0.8706	0.5176	4.1208
MWGF^［19］	4.8047	50.6551	61.1096	0.8796	0.5171	4.0131
MST_SR^［18］	4.7902	50.4261	60.9614	0.8715	0.5091	3.7675
DSIFT^［4］	4.8474	51.0602	61.0577	0.8688	0.5181	4.1049
ECNN^［13］	4.7996	50.5505	60.9748	0.8515	0.5132	4.1329
IFCNN^［12］	4.7674	50.0583	60.5222	0.8357	0.4997	3.7195
CSR^［20］	3.5794	38.4410	60.1693	0.7556	0.3822	4.0791
GF^［17］	4.3566	45.9536	60.4738	0.8073	0.4936	3.9264
本文	4.8495	51.0874	61.0834	0.8802	0.5187	4.1213

指标	结构1	结构2	结构3	结构4
AG	6.8893	6.9237	6.9333	6.9459
EI	71.2934	71.9763	71.9877	72.0919
STD	64.2943	64.3765	64.3822	64.4815
VIF	0.9239	0.9333	0.9302	0.9444
Q^AB/F	0.7132	0.7142	0.7144	0.7153
MI	5.3233	5.4456	5.4537	5.6633

loss	函数1	函数2	函数3	函数4
AG	6.9234	6.9322	6.9389	6.9459
EI	71.4843	71.9832	71.9899	72.0919
STD	64.3768	64.4576	64.4638	64.4815
VIF	0.9322	0.9356	0.9364	0.9444
Q^AB/F	0.7128	0.7136	0.7145	0.7153
MI	5.4323	5.4538	5.4649	5.6633

Multi⁃focus image fusion algorithm based on pixel⁃level convolutional neural network

RICH HTML

PDF (PC)