吉林大学学报(工学版) ›› 2019, Vol. 49 ›› Issue (5): 1726-1734.doi: 10.13229/j.cnki.jdxbgxb20180366

• • 上一篇    

基于残差连接卷积神经网络的图像超分辨率重构

郭继昌(),吴洁,郭春乐,朱明辉   

  1. 天津大学 电气自动化与信息工程学院,天津 300072
  • 收稿日期:2018-04-20 出版日期:2019-09-01 发布日期:2019-09-11
  • 作者简介:郭继昌(1966-),男,教授,博士.研究方向:图像处理.E-mail:jcguo@tju.edu.cn
  • 基金资助:
    国家自然科学基金项目(61771334)

Image super-resolution reconstruction based on residual connection convolutional neural network

Ji-chang GUO(),Jie WU,Chun-le GUO,Ming-hui ZHU   

  1. School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China
  • Received:2018-04-20 Online:2019-09-01 Published:2019-09-11

摘要:

针对基于卷积神经网络超分辨率重构算法中存在的感受野较小、梯度信息易丢失与网络收敛较慢等问题,提出了基于残差连接卷积神经网络的图像超分辨率重构算法。通过在低分辨率空间进行图像的超分辨率重构,减少了图像预处理过程,降低了网络复杂度。利用局部和全局残差连接,对卷积网络结构和亚像素采样层进行改进,局部残差促进了网络中信息的流动,全局残差使网络只学习图像残差信息,减少了网络冗余。通过增加网络深度扩大了感受野,使网络学习到更多的重建信息。实验结果表明:本文算法的PSNR和SSIM值相较于其他算法有不同程度的提升。

关键词: 信息处理技术, 图像超分辨率重构, 卷积神经网络, 亚像素采样层, 残差连接

Abstract:

Super-resolution reconstruction algorithms based on convolutional neural network are confront the problems of small receptive field, disappearing of gradient information and slow convergence of the network. In order to solve these problems, an image super-resolution reconstruction algorithm based on residual connection convolutional neural network is proposed. Processing images in high resolution space results in the increase in network complexity. So, images are processed in low-resolution space to reduce complexity. The convolutional neural network is improved by introducing local and global residuals. Local residual promotes the flow of information and avoids the problem of gradient disappearing during network training. Due to the local residual connection, image information can be transferred to deeper layers. The global residual makes the network learn the residual information in images only, reducing network redundancy and accelerating network convergence. The receptive field is expanded by increasing the network depth, which makes the network learn more reconstruction information. The PSNR and SSIM of the proposed algorithm are comparable to other algorithms.

Key words: imformation processing technology, image super-resolution, convolution neural network, subpixel upsampling, residual connection

中图分类号: 

  • TP751.1

图1

基于残差连接的卷积神经网络结构"

图2

残差单元"

图3

使用卷积的亚像素采样层"

图4

Set5测试集上PSNR收敛曲线"

表1

本文算法与ESPCN的PSNR值比较"

数据集 ESPCN(91) 本文算法(91)

Set5

Set14

BSD300

BSD500

平均值

32.39

28.97

28.20

28.27

29.46

32.56

29.01

28.31

28.40

29.58

图5

网络深度对PSNR值、复杂度和速度的影响"

表2

Set5测试集上不同算法重建图像的PSNR结果"

图像 双三次插值 A+ SRCNN FSRCNN 本文方法
Baby 37.07 38.37 38.36 38.40 38.55
Bird 36.81 41.01 40.46 40.80 41.82

Butterfly

Head

Women

27.43

34.86

32.14

32.02

35.74

35.27

32.14

35.67

34.83

32.64

35.66

35.36

33.72

35.89

35.70

图6

Set14中Baboon重建对比图"

图7

Set14中Face重建对比图"

图8

Set14中Foreman重建对比图"

图9

Set14中Pepper重建对比图"

表3

不同算法在放大因子为2、3、4时在Set5、Set14和BSD100测试集上的平均PSNR/SSIM"

数据库 尺度因子 双三次插值PSNR/SSIM

A+

PSNR/SSIM

SRCNN

PSNR/SSIM

FSRCNN

PSNR/SSIM

本文方法PSNR/SSIM
Set5 ×2 33.66/0.992 36.54/0.954 36.66/0.954 37.00/0.955 37.13/0.956
Set5 ×3 30.39/0.868 32.58/0.908 32.75/0.909 33.16/0.914 33.12/0.912
×4 28.42/0.810 30.28/0.860 30.49/0.862 30.70/0.865 30.74/0.865
Set14 ×2 30.24/0.868 32.28/0.905 32.42/0.906 32.63/0.909 32.72/0.917
×3 27.55/0.774 29.13/0.818 29.28/0.820 29.43/0.824 29.43/0.837
×4 26.00/0.702 27.32/0.749 27.49/0.750 27.59/0.754 27.60/0.772
×2 29.56/0.843 31.21 /0.886 31.36/0.888 31.50/0.891 31.69/0.899
BSD100 ×3 27.21/0.738 28.29/0.784 28.41/0.786 28.52/0.789 28.57/0.793
×4 25.96/0.667 26.82/0.709 26.90/0.710 26.96/0.713 27.07/0.717
平均值 - 28.78/0.807 30.98/0.841 30.64/0.843 31.91/0.873 31.97/0.878
1 Tsai R Y , Huang T S . Multiframe image restoration and registration[J]. Advance Computer Visual and Image Processing, 1984, 1(2): 317-339.
2 Peleg S , Keren D , Schweitzer L . Improving image resolution using subpixel motion[J]. Pattern Recognition Letters, 1987, 5(3): 223-226.
3 Schultz R R , Stevenson R L . Extraction of high-resolution frames from video sequences[J]. IEEE Transactions on Image Processing, 1996, 5(6): 996-1011.
4 Freeman W T , Jones T R , Pasztor E C . Example-based super-resolution[J]. IEEE Computer Graphics & Applications, 2002, 22(2): 56-65.
5 薛翠红, 于明, 杨宇皓, 等 . 基于学习的马尔科夫超分辨率复原[J]. 吉林大学学报: 工学版, 2013, 43(增刊1): 406-409.
Xue Cui-hong , Yu Ming , Yang Yu-hao , et al . MRF reconstruction based on Markov network[J]. Journal of Jilin University (Engineering and Technology Edition), 2013, 43(Sup.1): 406-409.
6 徐岩, 孙美双 . 基于卷积神经网络的水下图像增强方法[J]. 吉林大学学报: 工学版, 2018, 48(6): 1895-1903.
Xu Yan , Sun Mei-shuang . Enhancing underwater image based on convolutional neural networks[J]. Journal of Jilin University (Engineering and Technology Edition), 2018, 48(6): 1895-1903.
7 马淼, 李贻斌 . 基于多级图像序列和卷积神经网络的人体行为识别[J]. 吉林大学学报: 工学版, 2017, 47(4): 1244-1252.
Ma Miao , Li Yi-bin . Multi-level image sequences and convolutional neural networks based human action recognition method[J]. Journal of Jilin University (Engineering and Technology Edition), 2017, 47(4): 1244-1252.
8 Dong C , Chen C L , He K , et al . Learning a deep convolutional network for image super-resolution[C]∥European Conference on Computer Vision. Cham: Springer International Publishing, 2014: 184-199.
9 Dong C , Loy C C , Tang X . Accelerating the super-resolution convolutional neural network[C]∥European Conference on Computer Vision. Amsterdam: Springer International Publishing, 2016: 391-407.
10 Shi W , Caballero J , Huszár F , et al . Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]∥IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE Computer Society, 2016: 1874-1883.
11 Kim J , Lee J K , Lee K M . Accurate image super-resolution using very deep convolutional networks[C]∥IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE Computer Society, 2016: 1646-1654.
12 Kim J , Kwon Lee J , Mu Lee K . Deeply-recursive convolutional network for image super-resolution[C]∥IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE Computer Society, 2016: 1637-1645.
13 Justin J , Alexandre A , Li F F . Perceptual losses for real-time style transfer and super-resolution[C]∥European Conference on Computer Vision. Amsterdam: Springer International Publishing, 2016: 694-711.
14 Ledig C , Wang Z , Shi W , et al . Photo-realistic single image super-resolution using a generative adversarial network[C]∥IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE Computer Society, 2017: 105-114.
15 Cui Z , Chang H , Shan S , et al . Deep network cascade for image super-resolution[C]∥European Conference on Computer Vision. Cham: Springer International Publishing, 2014: 49-64.
16 Mao X J , Shen C , Yang Y B . Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[J]. Advances in Neural Information Processing Systems, 2016(2): 2810-2818.
17 Wang Y F , Wang L J , Wang H Y , et al . End-to-end image super-resolution via deep and shallow convolutional networks[J]. IEEE Access, 2016(7): 959-970.
18 He K , Zhang X , Ren S , et al . Deep residual learning for image recognition[C]∥IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE Computer Society, 2016: 770-778.
19 Nah S , Kim T H , Lee K M . Deep multi-scale convolutional neural network for dynamic scene deblurring[C]∥IEEE Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE Computer Society, 2017: 257-265.
[1] 卢洋,王世刚,赵文婷,赵岩. 基于离散Shearlet类别可分性测度的人脸表情识别方法[J]. 吉林大学学报(工学版), 2019, 49(5): 1715-1725.
[2] 董超,刘晶红,徐芳,王仁浩. 光学遥感图像舰船目标快速检测方法[J]. 吉林大学学报(工学版), 2019, 49(4): 1369-1376.
[3] 王柯俨,胡妍,王怀,李云松. 结合天空分割和超像素级暗通道的图像去雾算法[J]. 吉林大学学报(工学版), 2019, 49(4): 1377-1384.
[4] 托乎提努尔,张海龙,王杰,王娜,冶鑫晨,王万琼. 基于图形处理器的高速中值滤波算法[J]. 吉林大学学报(工学版), 2019, 49(3): 979-985.
[5] 付银娟,李勇,徐丽琴,张昆辉. NLFM⁃Costas射频隐身雷达信号设计及分析[J]. 吉林大学学报(工学版), 2019, 49(3): 994-999.
[6] 苏寒松,代志涛,刘高华,张倩芳. 结合吸收Markov链和流行排序的显著性区域检测[J]. 吉林大学学报(工学版), 2018, 48(6): 1887-1894.
[7] 徐岩,孙美双. 基于卷积神经网络的水下图像增强方法[J]. 吉林大学学报(工学版), 2018, 48(6): 1895-1903.
[8] 李居朋,张祖成,李墨羽,缪德芳. 基于Kalman滤波的电容屏触控轨迹平滑算法[J]. 吉林大学学报(工学版), 2018, 48(6): 1910-1916.
[9] 黄勇,杨德运,乔赛,慕振国. 高分辨合成孔径雷达图像的耦合传统恒虚警目标检测[J]. 吉林大学学报(工学版), 2018, 48(6): 1904-1909.
[10] 应欢,刘松华,唐博文,韩丽芳,周亮. 基于自适应释放策略的低开销确定性重放方法[J]. 吉林大学学报(工学版), 2018, 48(6): 1917-1924.
[11] 陆智俊,钟超,吴敬玉. 星载合成孔径雷达图像小特征的准确分割方法[J]. 吉林大学学报(工学版), 2018, 48(6): 1925-1930.
[12] 刘仲民,王阳,李战明,胡文瑾. 基于简单线性迭代聚类和快速最近邻区域合并的图像分割算法[J]. 吉林大学学报(工学版), 2018, 48(6): 1931-1937.
[13] 单泽彪,刘小松,史红伟,王春阳,石要武. 动态压缩感知波达方向跟踪算法[J]. 吉林大学学报(工学版), 2018, 48(6): 1938-1944.
[14] 姚海洋, 王海燕, 张之琛, 申晓红. 双Duffing振子逆向联合信号检测模型[J]. 吉林大学学报(工学版), 2018, 48(4): 1282-1290.
[15] 全薇, 郝晓明, 孙雅东, 柏葆华, 王禹亭. 基于实际眼结构的个性化投影式头盔物镜研制[J]. 吉林大学学报(工学版), 2018, 48(4): 1291-1297.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!