基于高效门控与目标区域关注的实时视频超分辨率

doi:10.13229/j.cnki.jdxbgxb.20240926

Abstract

Abstract:

Facing complex video scenes with large motion amplitude， real-time video super-resolution algorithms are difficult to reconstruct texture details and occluded regions， based on generative adversarial network， a real-time video super-resolution method based on efficient gating and target region attention is proposed. The method firstly uses an efficient gating reconstruction network as a generative network to maintain high efficiency while adaptively selecting complex region information through a simplified gating mechanism to enhance the reconstruction results. Further， the method proposes a target region attention discriminative network to provide multiscale and spatio-temporal information feedback for the generative network， and acquires multiscale information of the complex video through the multiscale mechanism and ReLU linear attention； through the significant spatio-temporal discriminative module， it restricts the discriminative network to focus on the complex regions， and better acquires the spatio-temporal information of the complex regions. The experimental results show that the proposed method exhibits significant superiority over other SOTA algorithms， the model efficiency achieves an inference delay of 13.36 ms and a real-time score of 65.806， which indicates the efficient real-time performance of the model.

Key words: video super-resolution, real-time, generative adversarial network, gating mechanisms, target region attention

CLC Number:

TP391.4

Le-ping LIN,Zhi SU,Ning OUYANG. Efficient gating and target region attention based real-time video super-resolution[J].Journal of Jilin University(Engineering and Technology Edition), 2026, 56(3): 819-829.

Figures/Tables 14

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Table 2

Fig.5

Fig.6

Fig.7

Table 3

Table 4

Table 5

Table 6

Table 7

References 23

[1]	Claudio Rota, Marco Buzzelli, Simone Bianco, et al. Video restoration based on deep learning: a comprehensive survey[J]. Artificial Intelligence Review, 2023, 56(6): 5317-5364.
[2]	Caballero J, Ledig C, Aitken A P, et al. Real-time video super-resolution with spatio-temporal networks and motion compensation[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4778-4787.
[3]	Vemulapalli R, Brown M, Mehdi S M. Frame-recurrent video super-resolution[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA,2018: 6626-6634.
[4]	Wang L, Guo Y, Liu L, et al. Deep video super-resolution using HR optical flow estimation[J].IEEE Transactions on Image Processing, 2020, 29: 4323-4336.
[5]	Chan K C K, Wang X, Yu K, et al. Basicvsr: the search for essential components in video super-resolution and beyond[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, TN, USA, 2021: 4947-4956.
[6]	Chan K C K, Zhou S, Xu X,et al. Basicvsr++: improving video super-resolution with enhanced propagation and alignment[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA,2022: 5972-5981.
[7]	Ouyang Ning, Zhi-shan Ou, Lin Le-ping. Video super-resolution network with gated high-low resolution frames[J]. Applied Sciences, 2023, 13(14): 1-16.
[8]	Zhou X, Zhang L, Zhao X,et al. Video super-resolution transformer with masked inter & intra-frame attention[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024: 25399-25408.
[9]	Cao Y, Wang C, Song C, et al. Real-time super-resolution system of 4k-video based on deep learning[C]∥2021 IEEE 32nd International Conference on Application-Specific Systems, Architectures and Processors(ASAP), NJ, USA, 2021: 69-76
[10]	Xia Bin, He Jing-wen, Zhang Yu-lun, et al. Structured sparsity learning for efficient video super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada. 2023: 22638-22647
[11]	Xiao Jun, Jiang Xin-yang, Zheng Ning-xin, et al.Online video super-resolution with convolutional kernel bypass grafts[J]. IEEE Transactions on Multimedia, 2023, 25: 8972-8987.
[12]	Li Gen, Ji Jie, Qin Ming-hai, et al. Towards high-quality and efficient video super-resolution via spatial-temporal data overfitting[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Vancouver, Canada. 2023: 10259-10269.
[13]	Chu Meng-yu, Xie You, Laura Leal-Taixé, et al. Temporally coherent gans for video super-resolution (tecogan)[J]. arXiv Preprint, 2018, 1(2): 3.
[14]	Chen Rui, Mu Yang, Zhang Yan. High-order relational generative adversarial network for video super-resolution[J]. Pattern Recognition, 2024, 146: 110059.
[15]	Chen L, Chu X, Zhang X,et al. Simple baselines for image restoration[C]∥European Conference on Computer Vision. Cham: Springer, 2022: 17-33.
[16]	Ding X, Zhang X, Han J, et al. Diverse branch block: Building a convolution as an inception-like unit[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, TN, USA, 2021: 10881-10890,.
[17]	Cai H, Li J, Gan C, et al. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Paris, France. 2023: 17302-17313.
[18]	Ouyang D, He S, Zhan J, et al. Efficient multi-scale attention module with cross-spatial learning[C]∥ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). ArXiv, 2023, abs/2305.13563.
[19]	Xue Tian-fan, Chen Bai-an, Wu Jia-jun, et al. Video enhancement with task-oriented flow[J]. International Journal of Computer Vision, 2019, 127: 1106-1125.
[20]	Liu Ce, Sun De-qing. On Bayesian adaptive video super resolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 36(2): 346-360.
[21]	Yi P, Wang Z, Jiang K,et al. Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations[C]∥Proceedings of the IEEE/CVF International Conference on Computer Vision,Seoul, Korea (South), 2019: 3106-3115.
[22]	Nah S, Baik S, Hong S,et al. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Long B, CA, USA, 2019, 1996-2005.
[23]	Ignatov A, Romero A, Kim H,et al. Real-time video super-resolution on smartphones with deep learning, Mobile AI2021 challenge: Report[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 2021: 2535-2544.

Related Articles 15

[1]	Liu ZHANG,Jia-xin LIANG,He LIU,Gui-xiang ZHANG,Wei LIU,Yan LI,Jia-bao ZHANG. Region of interest star map compression method based on real-time star location [J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(3): 802-810.
[2]	Lin LIN,Yu-xin CHEN,Wei-zhi NAI. Adaptive real⁃time gesture classification algorithm based on gesture frame sequence extraction [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(9): 3042-3048.
[3]	Yan PIAO,Ji-yuan KANG. RAUGAN：infrared image colorization method based on cycle generative adversarial networks [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(8): 2722-2731.
[4]	Hong XIAO,Xian-de LIU. Real-time acquisition and dynamic analysis of learning state based on hybrid intelligence [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(7): 2402-2408.
[5]	Jing-hua ZHAO,Da LIU,Yu-qi ZHOU,Long WEN,Qian-yu LIU,Jie LIU,Fang-xi XIE. Air⁃fuel ratio control of engines based on Gaussian process regression intake prediction [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(6): 1854-1861.
[6]	Guang-wen LIU,Qi-ying ZHAO,Chao WANG,Lian-yu Gao,Hua CAI,Qiang FU. Progressive recursive generative adversarial network-based single-image rain removal algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(4): 1363-1373.
[7]	Yuan JI,Ya-qi YU. Optimization algorithm for speech facial video generation based on dense convolutional generative adversarial networks and keyframes [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(3): 986-992.
[8]	Li-min YAN,Wei-ye JIN. Fast algorithm based on block direction prediction for AV1 intra encoding [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(3): 993-1000.
[9]	Hong ZHAO,Yu-xuan MA,Fu-rong SONG. Image adversarial examples generation based on Diff⁃AdvGAN [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(12): 4052-4062.
[10]	Xiang-long LUO,Xin-yu WEI,Mao-jun ZHAO,Ruo-chen LIU. Image dehazing algorithm based on contrast learning and generative adversarial network [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(10): 3296-3308.
[11]	Xiao-yue WEN,Guo-min QIAN,Hua-hua KONG,Yue-jie MIU,Dian-hai WANG. TrafficPro： a framework to predict link speeds on signalized urban traffic network [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(8): 2214-2222.
[12]	Xin-gang GUO,Ying-chen HE,Chao CHENG. Noise-resistant multistep image super resolution network [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(7): 2063-2071.
[13]	Yu-kun ZHENG,Ru-yue SUN,Feng-ming LI,Yi-xiang LIU,Dong-guang LI,Rui SONG. Research and application of the centralized drive and control system for a hydraulic manipulator [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(11): 3358-3371.
[14]	Feng-feng ZHOU,Tao YU,Yu-si FAN. Generative adversarial autoencoder integrated voting algorithm based on mass spectral data [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(10): 2969-2977.
[15]	Nan ZHANG,Jian-hua SHI,Ji YI,Ping WANG. Real⁃time tracking method of underground moving target based on weighted centroid positioning [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(5): 1458-1464.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

名称	环境配置
操作系统	Ubuntu20.04
处理器	Intel（R） Xeon（R） Gold6248 CPU @ 2.50 GHz
显卡	Tesla A100
CUDA版本	11.6
深度学习框架	PyTorch 1.13.0
Python版本	3.9
显存	80 GB

方法	REDS4			Vid4			UDM10
方法	PSNR/dB↑	SSIM↑	LPIPS↓	PSNR/dB↑	SSIM↑	LPIPS↓	PSNR/dB↑	SSIM↑	LPIPS↓
VESPCN	22.29	0.594 4	0.334 2	19.70	0.500 2	0.574 7	26.22	0.791 6	0.143 6
SOFVSR	24.31	0.660 2	0.317 2	21.07	0.530 3	0.356 8	27.63	0.816 9	0.131 1
TecoGAN	22.99	0.630 3	0.291 6	24.81	0.770 7	0.200 9	31.97	0.900 9	0.088 8
FRVSR	24.43	0.684 9	0.333 8	25.55	0.798 5	0.280 5	33.95	0.925 6	0.099 5
EGVSR	29.75	0.838 2	0.109 0	25.83	0.796 3	0.137 6	36.96	0.941 1	0.067 6
STDO	29.34	0.844 1	0.189 8	26.30	0.790 2	0.267 6	38.23	0.958 9	0.084 0
SSL	29.81	0.835 6	0.223 7	26.61	0.819 6	0.245 3	38.26	0.957 2	0.079 6
EGTRA	29.89	0.839 9	0.085 1	26.75	0.822 0	0.134 8	38.69	0.966 1	0.050 3

方法	模型参数	FLOPs	推理延迟/ms	实时推理	REDS4评分	REDS4 PSNR
VESPCN	0.879M	96.56G	20.62	√	0.001	22.29
SOFVSR	1.640M	226.12G	75.13	×	0.001	24.31
TecoGAN	2.589M	190.81G	32.10	√	0.001	22.99
FRVSR	2.589M	190.81G	32.09	√	0.014	24.43
EGVSR	2.681M	102.89G	14.25	√	50.812	29.75
STDO	4.843M	2.76G	35.71	√	11.485	29.34
SSL	0.589M	63.9G	18.25	√	43.117	29.81
EGTRA	2.177M	60.69G	13.36	√	65.806	29.89

方法	3×3 支路	5×5 支路	REDS4			UDM10			Vid4
方法	3×3 支路	5×5 支路	PSNR/dB↑	SSIM↑	LPIPS↓	PSNR/dB↑	SSIM↑	LPIPS↓	PSNR/dB↑	SSIM↑	LPIPS↓
Model l			29.59	0.832 2	0.107 6	26.54	0.816 2	0.144 8	38.48	0.962 3	0.079 5
Model 2	√		29.68	0.834 1	0.095 0	26.61	0.816 3	0.142 9	38.54	0.963 3	0.058 3
Model 3		√	29.64	0.834 8	0.097 4	26.66	0.818 9	0.138 0	38.56	0.963 8	0.069 2
Model 4	√	√	29.89	0.839 9	0.085 1	26.75	0.822 0	0.134 8	38.69	0.966 1	0.050 3

方法	EGRN	RLA	SSTDM	REDS4			UDM10			Vid4
方法	EGRN	RLA	SSTDM	PSNR/ dB↑	SSIM↑	LPIPS↓	PSNR/ dB↑	SSIM↑	LPIPS↓	PSNR/ dB↑	SSIM↑	LPIPS↓
SSTDM-o	√	√		29.86	0.838 1	0.089 3	26.71	0.819 8	0.143 9	38.59	0.964 9	0.064 0
RLA-o	√		√	29.59	0.832 2	0.107 6	26.54	0.816 2	0.144 8	38.48	0.964 1	0.079 5
EGRN-o		√	√	29.41	0.830 8	0.085 2	26.29	0.814 2	0.135 1	38.20	0.960 5	0.050 8
Model	√	√	√	29.89	0.839 9	0.085 1	26.75	0.822 0	0.134 8	38.69	0.966 1	0.050 3

Efficient gating and target region attention based real-time video super-resolution

RICH HTML

PDF (PC)