Journal of Jilin University(Engineering and Technology Edition) ›› 2025, Vol. 55 ›› Issue (8): 2722-2731.doi: 10.13229/j.cnki.jdxbgxb.20240012

Previous Articles    

RAUGAN：infrared image colorization method based on cycle generative adversarial networks

Yan PIAO(),Ji-yuan KANG   

  1. School of Electronic and Information Engineering，Changchun University of Science and Technology，Changchun 130022，China
  • Received:2024-01-04 Online:2025-08-01 Published:2025-11-14

RICH HTML

 0   

PDF (PC)

9

Abstract:

In order to solve the problems of color distortion， semantic ambiguity， unclear texture and shape characteristics in the process of near-infrared image colorization， we propose a method of infrared image colorization （RAUGAN） . Firstly， the CycleGAN network generator is improved， and a Res-ASPP-UNet network is designed and fused， which connects atrous spatial pyramid pooling（ASPP） to Skip connection structure of the original UNet， the output characteristic graphs of different scales in the decoding branch can be combined with the corresponding output characteristic graphs in the encoder. Secondly， a deep bottleneck layer composed of residual block， convolutional block attention module （CBAM） is designed to replace the bottleneck layer in UNET network to enhance the local area feature， improve its recognition ability and prevent gradient explosion. Finally， the perceptual loss function is introduced in the discriminant network to solve the problem of color restoration distortion.Experimental results show that the proposed method is superior to the original CycleGAN network.

Key words: computer application, infrared image colorization, cycle-consistent generative adversarial networks, atrous spatial pyramid pooling, attention mechanism

CLC Number: 

  • TP391.41

Fig.1

Overall framework of the network"

Fig.2

Our Improved generator network framework"

Table 1

Generator network configuration"

network layerOutputConvolution（Conv）、stride（s）、Padding（p）
Input256×256
ConvBlock 1256×2563×3ConvBatchNormReLU，s=1，p=1
DBlock 164×643×3ConvBatchNormLeakyReLU，s=2，p=1
DBlock 2125×1283×3ConvBatchNormLeakyReLU，s=2，p=1
DBlock 3256×2563×3ConvBatchNormLeakyReLU，s=2，p=1
Output256×2563×3ConvBatchNormReLU，s=2，p=2

Fig.3

Network architecture of ASPP model"

Fig.4

Network architecture of deep bottleneck layer block"

Fig.5

Residual block network structure"

Fig.6

Structure of the CBAM module"

Fig.7

Discriminator model"

Fig.8

Comparison results of each algorithm for the KAIST dataset"

Fig.9

Comparison results of each algorithm for the IRVI dataset"

Table 2

Comparison of objective evaluation indexes of each image in Fig. 8"

ImageEvaluation indicatorCycleGANCDGANI2VGANMCLOurs
Group 1

PSNR/dB

SSIM

19.80

0.72

26.18

0.85

29.45

0.90

28.84

0.93

33.80

0.96

Group 2

PSNR/dB

SSIM

17.38

0.84

20.25

0.86

22.82

0.89

24.03

0.90

27.52

0.92

Group 3

PSNR/dB

SSIM

22.50

0.83

25.87

0.90

25.67

0.94

28.05

0.94

31.29

0.95

Group 4

PSNR/dB

SSIM

21.21

0.78

24.21

0.90

25.70

0.91

27.38

0.92

28.03

0.94

Group 5

PSNR/dB

SSIM

19.42

0.77

23.93

0.89

24.02

0.90

25.76

0.93

27.86

0.93

Group 6

PSNR/dB

SSIM

22.31

0.80

23.11

0.87

25.24

0.92

27.34

0.91

29.00

0.93

Table 3

Comparison of objective evaluation indexes of each image in Fig. 9"

ImageEvaluation indicatorCycleGANCDGANI2VGANMCLOurs
Group 1

PSNR/dB

SSIM

22.32

0.88

27.55

0.92

29.27

0.93

30.27

0.94

32.97

0.96

Group 2

PSNR/dB

SSIM

22.22

0.67

25.33

0.87

24.89

0.88

26.68

0.90

29.99

0.93

Group 3

PSNR/dB

SSIM

23.22

0.88

29.26

0.90

29.46

0.93

31.35

0.95

31.73

0.96

Group 4

PSNR/dB

SSIM

21.49

0.88

29.61

0.87

29.48

0.91

31.65

0.93

32.09

0.97

Group 5

PSNR/dB

SSIM

19.55

0.83

30.50

0.91

29.10

0.89

30.46

0.90

33.19

0.97

Group 6

PSNR/dB

SSIM

22.17

0.82

27.76

0.87

28.10

0.90

29.03

0.93

31.68

0.95

Table 4

Comparison of evaluation indexes for colorization of NIR images"

算法PSNR/dBSSIM
CycleGAN27.640.82
CycleGAN+ASPP31.340.91
CycleGAN+CBAM29.650.87
CycleGAN+ASPP+CBAM33.270.93
CycleGAN+ASPP+CBAM+感知损失函数33.120.96

Fig.10

Learned perceptual image patch similarity trend before and after improvement"

[1] Reinhard E, Adhikhmin M, Gooch B, et al. Color transfer between images[J]. IEEE Computer Graphics and Applications, 2001, 21(5): 34-41.
[2] Welsh T, Ashikhmin M, Mueller K. Transferring color to greyscale images[J]. ACM Transactions on Graphics, 2002, 21(3): 277-280.
[3] Levin A, Lischinski D, Weiss Y. Colorization using optimization[J]. ACM Transactions on Graphics, 2004, 23(3): 689-694.
[4] Goodfellow I J, Pouget A J, Mirza M, et al.Generative adversarial nets[C]∥Advances in Neural Information Processing Systems, Montréal, Canada, 2014:2672-2680.
[5] Isola P, Zhu J, Zhou T, et al. Image-to-image translation with conditional adversarial networks[C]∥Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, Las Vegas, USA, 2016: 1125-1134.
[6] Park T, Liu M Y, Wang T C, et al. Semantic imagesynthesis with spatially-adaptive normalization[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 2337-2346.
[7] Wang T C, Liu M Y, Zhu J Y, et al. High-resolutionimage synthesis and semantic manipulation withconditional gans[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 8798-8807.
[8] Huang X, Liu M Y, Belongie S, et al. Multimodal unsupervised image-to-image translation[C]∥Proceedings of the European Conference on Computer Vision, Munichi, Germany, 2018:172-189.
[9] Lee H Y, Tseng H Y, Huang J B, et al. Diverse image-to-image translation via disentangled representations[C]∥Proceedings of the European Conference on Computer Vision, Munichi, Germany, 2019:35-51.
[10] Yi Z, Zhang H, Tan P, et al. Dualgan: unsupervised dual learning for image-to-image translation[C]∥Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 2017: 2849-2857.
[11] Suárez P L, Sappa A D, Vintimilla B X. Infrared image colorization based on a triplet dcgan architecture[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, USA, 2017: 212-217.
[12] 王晓宇.基于颜色迁移的图像彩色化算法研究[D]. 长春: 长春理工大学电子信息工程学院, 2020.
Wang Xiao-yu. Research on image colorization algorithm based on color migration[D]. Changchun: School of Electronic and Information Engineering, Changchun University of Science and Technology, 2020.
[13] Li S, Han B F, Yu Z J, et al. I2VGAN:unpaired Infrared to-visible video translation[C]∥Proceedings of the 29th ACM International　Conference on　Multimedia, New York,USA, 2021:3061-3069.
[14] 高美玲, 段锦, 莫苏新, 等. 基于空洞循环卷积的近红外图像彩色化方法[J]. 光学技术, 2022, 48(6):742-748.
Gao Mei-ling, Duan Jin, Mo Su-xin, et al. Near infrared image colorization method based on dilated-cycle convolution[J]. Optical Technique, 2022, 48(6): 742-748.
[15] Chen J, Chen J, Chao H, et al. Image blind denoising with generative adversarial network based noise modeling[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018:3155-3164.
[16] 陈雪云, 许韬, 黄小巧. 基于条件生成对抗网络的医学细胞图像生成检测方法[J]. 吉林大学学报: 工学版, 2021, 51(4): 1414-1419.
Chen Xue-yun, Xu Tao, Huang Xiao-qiao. Detection method of medical cell image generation basedon conditional generative adversarial network[J]. Journal of Jilin University (Engineering and Technology Edition), 2021, 51(4): 1414-1419.
[17] 王小玉, 胡鑫豪, 韩昌林. 基于生成对抗网络的人脸铅笔画算法[J]. 吉林大学学报: 工学版, 2021, 51(1): 285-292.
Wang Xiao-yu, Hu Xin-hao, Han Chang-lin. Face pencil drawing algorithms based on generative adversarial network[J]. Journal of Jilin University (Engineering nd Technology Edition), 2021, 51(1): 285-292.
[18] Monday H N, Li J, Nneji G U, et al. A wavelet convolutional capsule network with modified super resolution generative adversarial network for fault diagnosis and classification[J]. Complex & Intelligent Systems, 2022, 8: 4831-4847.
[19] 彭晏飞, 张平甲, 高艺, 等. 融合注意力的生成式对抗网络单图像超分辨率重建[J]. 激光与光电子学进展, 2021, 58(20): 182-191.
Peng Yan-fei, Zhang Ping-jia, Gao Yi, et al. Attention fusion generative adversarial network for single-image super-resolution reconstruction[J]. Laser &Optoelectronics Progress, 2021, 58(20): 182-191.
[20] Liang C C, George P, Iasonas K, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
[21] Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module[C]∥Proceedings of the 15thEurope an Conference on Computer Visdom, Munich, Germany, 2018: 3-19.
[22] Li C, Wan D M. Precomputed real-time texture synthesis with markovian generative adversarial networks[C]∥European Conference on Computer Vision of IEEE, Amsterdam, The Netherlands, 2016:702-716.
[23] Wang S, Park J, Kim N, et al. Multispectral pedestrian detection: benchmark dataset and baseline[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015: 1037-1045.
[24] Park T, Liu M Y, Wang T C, et al. Semantic image synthesis with spatially-adaptive normalization[J]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019:2337-2346.
[25] Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial network[C]∥IEEE International Conference on Computer Vision, Venice, Italy, 2017: 2223-2232.
[26] Kancharagunta B K, Ram S D. CDGAN: cyclic discriminative generative adversarial networks forimage-to-image transformation[J]. Journal of Visual Communication and Image Representation, 2022, 82:103382.
[27] Gou Y. Multi-feature contrastive learning for unpaired image-to-image translation[J]. Intelligent Systems, 2023, 9: 4111-4122.
[1] Qing-lin AI,Yuan-xiao LIU,Jia-hao YANG. Small target swmantic segmentation method based MFF-STDC network in complex outdoor environments [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(8): 2681-2692.
[2] Yu-fei ZHANG,Li-min WANG,Jian-ping ZHAO,Zhi-yao JIA,Ming-yang LI. Robot inverse kinematics solution based on center selection battle royale optimization algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(8): 2703-2710.
[3] Qiong-xin LIU,Tian-tian WANG,Ya-nan WANG. Non-dominated sorted particle swarm genetic algorithm to solve vehicle location routing problems [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(7): 2464-2474.
[4] Xiang-jiu CHE,Liang LI. Graph similarity measurement algorithm combining global and local fine-grained features [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(7): 2365-2371.
[5] Wen-hui LI,Chen YANG. Few-shot remote sensing image classification based on contrastive learning text perception [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(7): 2393-2401.
[6] Shan-na ZHUANG,Jun-shuai WANG,Jing BAI,Jing-jin DU,Zheng-you WANG. Video-based person re-identification based on three-dimensional convolution and self-attention mechanism [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(7): 2409-2417.
[7] Zhi-gang FENG,Shou-qi WANG,Ming-yue YU. Rolling bearing fault diagnosis based on variational mode extraction and lightweight network [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 1883-1891.
[8] Jian WANG,Chen-wei JIA. Trajectory prediction model for intelligent connected vehicle [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 1963-1972.
[9] Hai-peng CHEN,Shi-bo ZHANG,Ying-da LYU. Multi⁃scale context⁃aware and boundary⁃guided image manipulation detection method [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 2114-2121.
[10] Feng-feng ZHOU,Zhe GUO,Yu-si FAN. Feature representation algorithm for imbalanced classification of multi⁃omics cancer data [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 2089-2096.
[11] Xiang-jiu CHE,Yu-peng SUN. Graph node classification algorithm based on similarity random walk aggregation [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 2069-2075.
[12] Ping-ping LIU,Wen-li SHANG,Xiao-yu XIE,Xiao-kang YANG. Unbalanced image classification algorithm based on fine⁃grained analysis [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 2122-2130.
[13] You-wei WANG,Ao LIU,Li-zhou FENG. New method for text sentiment classification based on knowledge distillation and comment time [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(5): 1664-1674.
[14] Hong-wei ZHAO,Ming-zhu ZHOU,Ping-ping LIU,Qiu-zhan ZHOU. Medical image segmentation based on confident learning and collaborative training [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(5): 1675-1681.
[15] Zi-hao SHEN,Yong-sheng GAO,Hui WANG,Pei-qian LIU,Kun LIU. Deep deterministic policy gradient caching method for privacy protection in Internet of Vehicles [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(5): 1638-1647.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright © Journal of Jilin University(Engineering and Technology Edition), All Rights Reserved.
Tel: +86-431-85095297　Fax: +86-431-85094074　E-mail: xbgxb@jlu.edu.cn
Powered by Beijing Magtech Co. Ltd