基于高效门控与目标区域关注的实时视频超分辨率

doi:10.13229/j.cnki.jdxbgxb.20240926

吉林大学学报(工学版) ›› 2026, Vol. 56 ›› Issue (3): 819-829.doi: 10.13229/j.cnki.jdxbgxb.20240926

• 计算机科学与技术 • 上一篇

基于高效门控与目标区域关注的实时视频超分辨率

林乐平^1,²(),苏治²,欧阳宁^1,²()

^1.桂林电子科技大学广西无线宽带通信与信号处理重点实验室，广西桂林 541004
^2.桂林电子科技大学信息与通信学院，广西桂林 541004

收稿日期:2024-08-23 出版日期:2026-03-01 发布日期:2026-03-31
通讯作者: 欧阳宁 E-mail:linleping@guet.edu.cn;ynou@guet.edu.cn
作者简介:林乐平（1980-），女，教授，博士.研究方向：机器学习，图像信号处理. E-mail：linleping@guet.edu.cn
基金资助:
国家自然科学基金项目(62001133);广西科技基地和人才专项项目(桂科AD19110060);广西自然科学基金项目(2017GXNSFBA198212);广西无线宽带通信与信号处理重点实验室基金项目(GXKL06200114)

Efficient gating and target region attention based real-time video super-resolution

Le-ping LIN^1,²(),Zhi SU²,Ning OUYANG^1,²()

^1.Guangxi Key Laboratory of Wireless Broadband Communication and Signal Processing，Guilin University of Electronic Technology，Guilin 541004，China
^2.School of Information and Communication，Guilin University of Electronic Technology，Guilin 541004，China

Received:2024-08-23 Online:2026-03-01 Published:2026-03-31
Contact: Ning OUYANG E-mail:linleping@guet.edu.cn;ynou@guet.edu.cn

摘要/Abstract

摘要：

面对大运动幅度的复杂视频场景，实时视频超分辨率算法难以重建纹理细节、遮挡区域。本文基于生成对抗网络，提出了一种基于高效门控与目标区域关注的实时视频超分辨率方法。该方法首先使用高效门控重建网络作为生成网络，在保持高效的同时，通过简化的门控机制自适应选择复杂区域信息，以提升重建结果。进一步地，该方法提出了目标区域关注鉴别网络，为生成网络提供多尺度及时空信息反馈，通过多尺度机制和ReLU线性注意力获取复杂视频的多尺度信息；通过显著时空鉴别模块，限制鉴别网络关注复杂区域，以更好地获取复杂区域的时空信息。实验结果表明，所提方法相较于其他先进算法具有显著的优越性；在模型效率方面，实现了13.36 ms的推理延迟及65.806的实时得分，显示了模型高效的实时性能。

关键词: 视频超分辨率, 实时, 生成对抗网络, 门控机制, 目标区域关注

Abstract:

Facing complex video scenes with large motion amplitude， real-time video super-resolution algorithms are difficult to reconstruct texture details and occluded regions， based on generative adversarial network， a real-time video super-resolution method based on efficient gating and target region attention is proposed. The method firstly uses an efficient gating reconstruction network as a generative network to maintain high efficiency while adaptively selecting complex region information through a simplified gating mechanism to enhance the reconstruction results. Further， the method proposes a target region attention discriminative network to provide multiscale and spatio-temporal information feedback for the generative network， and acquires multiscale information of the complex video through the multiscale mechanism and ReLU linear attention； through the significant spatio-temporal discriminative module， it restricts the discriminative network to focus on the complex regions， and better acquires the spatio-temporal information of the complex regions. The experimental results show that the proposed method exhibits significant superiority over other SOTA algorithms， the model efficiency achieves an inference delay of 13.36 ms and a real-time score of 65.806， which indicates the efficient real-time performance of the model.

Key words: video super-resolution, real-time, generative adversarial network, gating mechanisms, target region attention

中图分类号:

TP391.4

林乐平,苏治,欧阳宁. 基于高效门控与目标区域关注的实时视频超分辨率[J]. 吉林大学学报(工学版), 2026, 56(3): 819-829.

Le-ping LIN,Zhi SU,Ning OUYANG. Efficient gating and target region attention based real-time video super-resolution[J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(3): 819-829.

图/表 14

图1

图2

图3

图4

表1

表2

图5

图 6

图 7

表 3

表4

表5

表6

表7

参考文献 23

[1]	Claudio Rota, Marco Buzzelli, Simone Bianco, et al. Video restoration based on deep learning: a comprehensive survey[J]. Artificial Intelligence Review, 2023, 56(6): 5317-5364.
[2]	Caballero J, Ledig C, Aitken A P, et al. Real-time video super-resolution with spatio-temporal networks and motion compensation[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4778-4787.
[3]	Vemulapalli R, Brown M, Mehdi S M. Frame-recurrent video super-resolution[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA,2018: 6626-6634.
[4]	Wang L, Guo Y, Liu L, et al. Deep video super-resolution using HR optical flow estimation[J].IEEE Transactions on Image Processing, 2020, 29: 4323-4336.
[5]	Chan K C K, Wang X, Yu K, et al. Basicvsr: the search for essential components in video super-resolution and beyond[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, TN, USA, 2021: 4947-4956.
[6]	Chan K C K, Zhou S, Xu X,et al. Basicvsr++: improving video super-resolution with enhanced propagation and alignment[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA,2022: 5972-5981.
[7]	Ouyang Ning, Zhi-shan Ou, Lin Le-ping. Video super-resolution network with gated high-low resolution frames[J]. Applied Sciences, 2023, 13(14): 1-16.
[8]	Zhou X, Zhang L, Zhao X,et al. Video super-resolution transformer with masked inter & intra-frame attention[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024: 25399-25408.
[9]	Cao Y, Wang C, Song C, et al. Real-time super-resolution system of 4k-video based on deep learning[C]∥2021 IEEE 32nd International Conference on Application-Specific Systems, Architectures and Processors(ASAP), NJ, USA, 2021: 69-76
[10]	Xia Bin, He Jing-wen, Zhang Yu-lun, et al. Structured sparsity learning for efficient video super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada. 2023: 22638-22647
[11]	Xiao Jun, Jiang Xin-yang, Zheng Ning-xin, et al.Online video super-resolution with convolutional kernel bypass grafts[J]. IEEE Transactions on Multimedia, 2023, 25: 8972-8987.
[12]	Li Gen, Ji Jie, Qin Ming-hai, et al. Towards high-quality and efficient video super-resolution via spatial-temporal data overfitting[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Vancouver, Canada. 2023: 10259-10269.
[13]	Chu Meng-yu, Xie You, Laura Leal-Taixé, et al. Temporally coherent gans for video super-resolution (tecogan)[J]. arXiv Preprint, 2018, 1(2): 3.
[14]	Chen Rui, Mu Yang, Zhang Yan. High-order relational generative adversarial network for video super-resolution[J]. Pattern Recognition, 2024, 146: 110059.
[15]	Chen L, Chu X, Zhang X,et al. Simple baselines for image restoration[C]∥European Conference on Computer Vision. Cham: Springer, 2022: 17-33.
[16]	Ding X, Zhang X, Han J, et al. Diverse branch block: Building a convolution as an inception-like unit[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, TN, USA, 2021: 10881-10890,.
[17]	Cai H, Li J, Gan C, et al. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris, France. 2023: 17302-17313.
[18]	Ouyang D, He S, Zhan J, et al. Efficient multi-scale attention module with cross-spatial learning[C]∥ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). ArXiv, 2023, abs/2305.13563.
[19]	Xue Tian-fan, Chen Bai-an, Wu Jia-jun, et al. Video enhancement with task-oriented flow[J]. International Journal of Computer Vision, 2019, 127: 1106-1125.
[20]	Liu Ce, Sun De-qing. On Bayesian adaptive video super resolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 36(2): 346-360.
[21]	Yi P, Wang Z, Jiang K,et al. Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations[C]∥Proceedings of the IEEE/CVF International Conference on Computer Vision,Seoul, Korea (South), 2019: 3106-3115.
[22]	Nah S, Baik S, Hong S,et al. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long B, CA, USA, 2019, 1996-2005.
[23]	Ignatov A, Romero A, Kim H,et al. Real-time video super-resolution on smartphones with deep learning, Mobile AI2021 challenge: Report[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 2021: 2535-2544.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

名称	环境配置
操作系统	Ubuntu20.04
处理器	Intel（R） Xeon（R） Gold6248 CPU @ 2.50 GHz
显卡	Tesla A100
CUDA版本	11.6
深度学习框架	PyTorch 1.13.0
Python版本	3.9
显存	80 GB

方法	REDS4			Vid4			UDM10
方法	PSNR/dB↑	SSIM↑	LPIPS↓	PSNR/dB↑	SSIM↑	LPIPS↓	PSNR/dB↑	SSIM↑	LPIPS↓
VESPCN	22.29	0.594 4	0.334 2	19.70	0.500 2	0.574 7	26.22	0.791 6	0.143 6
SOFVSR	24.31	0.660 2	0.317 2	21.07	0.530 3	0.356 8	27.63	0.816 9	0.131 1
TecoGAN	22.99	0.630 3	0.291 6	24.81	0.770 7	0.200 9	31.97	0.900 9	0.088 8
FRVSR	24.43	0.684 9	0.333 8	25.55	0.798 5	0.280 5	33.95	0.925 6	0.099 5
EGVSR	29.75	0.838 2	0.109 0	25.83	0.796 3	0.137 6	36.96	0.941 1	0.067 6
STDO	29.34	0.844 1	0.189 8	26.30	0.790 2	0.267 6	38.23	0.958 9	0.084 0
SSL	29.81	0.835 6	0.223 7	26.61	0.819 6	0.245 3	38.26	0.957 2	0.079 6
EGTRA	29.89	0.839 9	0.085 1	26.75	0.822 0	0.134 8	38.69	0.966 1	0.050 3

方法	模型参数	FLOPs	推理延迟/ms	实时推理	REDS4评分	REDS4 PSNR
VESPCN	0.879M	96.56G	20.62	√	0.001	22.29
SOFVSR	1.640M	226.12G	75.13	×	0.001	24.31
TecoGAN	2.589M	190.81G	32.10	√	0.001	22.99
FRVSR	2.589M	190.81G	32.09	√	0.014	24.43
EGVSR	2.681M	102.89G	14.25	√	50.812	29.75
STDO	4.843M	2.76G	35.71	√	11.485	29.34
SSL	0.589M	63.9G	18.25	√	43.117	29.81
EGTRA	2.177M	60.69G	13.36	√	65.806	29.89

方法	3×3 支路	5×5 支路	REDS4			UDM10			Vid4
方法	3×3 支路	5×5 支路	PSNR/dB↑	SSIM↑	LPIPS↓	PSNR/dB↑	SSIM↑	LPIPS↓	PSNR/dB↑	SSIM↑	LPIPS↓
Model l			29.59	0.832 2	0.107 6	26.54	0.816 2	0.144 8	38.48	0.962 3	0.079 5
Model 2	√		29.68	0.834 1	0.095 0	26.61	0.816 3	0.142 9	38.54	0.963 3	0.058 3
Model 3		√	29.64	0.834 8	0.097 4	26.66	0.818 9	0.138 0	38.56	0.963 8	0.069 2
Model 4	√	√	29.89	0.839 9	0.085 1	26.75	0.822 0	0.134 8	38.69	0.966 1	0.050 3

方法	EGRN	RLA	SSTDM	REDS4			UDM10			Vid4
方法	EGRN	RLA	SSTDM	PSNR/ dB↑	SSIM↑	LPIPS↓	PSNR/ dB↑	SSIM↑	LPIPS↓	PSNR/ dB↑	SSIM↑	LPIPS↓
SSTDM-o	√	√		29.86	0.838 1	0.089 3	26.71	0.819 8	0.143 9	38.59	0.964 9	0.064 0
RLA-o	√		√	29.59	0.832 2	0.107 6	26.54	0.816 2	0.144 8	38.48	0.964 1	0.079 5
EGRN-o		√	√	29.41	0.830 8	0.085 2	26.29	0.814 2	0.135 1	38.20	0.960 5	0.050 8
Model	√	√	√	29.89	0.839 9	0.085 1	26.75	0.822 0	0.134 8	38.69	0.966 1	0.050 3

基于高效门控与目标区域关注的实时视频超分辨率

Efficient gating and target region attention based real-time video super-resolution

RICH HTML

PDF (PC)

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 23

相关文章 15

Metrics

本文评价

推荐阅读 0

[1]	张刘,梁嘉新,刘赫,张贵祥,刘威,李岩,章家保. 基于实时星点定位的感兴趣区域星图压缩方法[J]. 吉林大学学报(工学版), 2026, 56(3): 802-810.
[2]	林琳,陈雨欣,佴威至. 基于手势帧序列提取的自适应实时手势分类算法[J]. 吉林大学学报(工学版), 2025, 55(9): 3042-3048.
[3]	朴燕,康继元. RAUGAN：基于循环生成对抗网络的红外图像彩色化方法[J]. 吉林大学学报(工学版), 2025, 55(8): 2722-2731.
[4]	肖红,刘显德. 基于混合智能的学习状态实时采集与动态分析方法[J]. 吉林大学学报(工学版), 2025, 55(7): 2402-2408.
[5]	文斌,彭顺,杨超,沈艳军,李辉. 多深度自适应融合去雾生成网络[J]. 吉林大学学报(工学版), 2025, 55(6): 2103-2113.
[6]	赵靖华,刘妲,周宇麒,闻龙,刘倩妤,刘捷,解方喜. 基于高斯过程回归进气量预测的空燃比控制[J]. 吉林大学学报(工学版), 2025, 55(6): 1854-1861.
[7]	刘广文,赵绮莹,王超,高连宇,才华,付强. 基于渐进递归的生成对抗单幅图像去雨算法[J]. 吉林大学学报(工学版), 2025, 55(4): 1363-1373.
[8]	季渊,虞雅淇. 基于密集卷积生成对抗网络与关键帧的说话人脸视频生成优化算法[J]. 吉林大学学报(工学版), 2025, 55(3): 986-992.
[9]	严利民,金炜烨. 基于块方向预测的AV1快速帧内编码算法[J]. 吉林大学学报(工学版), 2025, 55(3): 993-1000.
[10]	赵宏,马宇轩,宋馥荣. 基于Diff-AdvGAN的图像对抗样本生成方法[J]. 吉林大学学报(工学版), 2025, 55(12): 4052-4062.
[11]	罗向龙,魏欣语,赵茂军,刘若辰. 融合对比学习和生成对抗网络的图像去雾算法[J]. 吉林大学学报(工学版), 2025, 55(10): 3296-3308.
[12]	张曦,库少平. 基于生成对抗网络的人脸超分辨率重建方法[J]. 吉林大学学报(工学版), 2025, 55(1): 333-338.
[13]	温晓岳,钱国敏,孔桦桦,缪月洁,王殿海. TrafficPro：一种针对城市信控路网的路段速度预测框架[J]. 吉林大学学报(工学版), 2024, 54(8): 2214-2222.
[14]	赖丹晖,罗伟峰,袁旭东,邱子良. 复杂环境下多模态手势关键点特征提取算法[J]. 吉林大学学报(工学版), 2024, 54(8): 2288-2294.
[15]	郭昕刚,何颖晨,程超. 抗噪声的分步式图像超分辨率重构算法[J]. 吉林大学学报(工学版), 2024, 54(7): 2063-2071.