基于增强对象学习和注意力网络的视频描述方法

doi:10.13229/j.cnki.jdxbgxb.20240251

摘要/Abstract

摘要：

在视频描述任务中，常见问题之一是对象描述不够具体，主要原因在于模型没有充分学习视频中的对象信息。同时，视频包含了丰富的特征信息，如对象信息、运动信息和上下文信息，这使得如何提升模型在生成描述时学习关键信息的能力成为一项具有挑战性的任务。为解决上述问题，本文提出了一种基于增强对象学习和注意力网络的方法。首先，设计了一种新的增强对象学习模块，旨在充分学习视频中的对象信息，从而实现对视频内容的准确描述；其次，构建了一种注意力网络，致力于有效关注不同类型的信息，以提升模型在生成描述时学习关键信息的能力。在MSVD和MSR-VTT数据集上的实验中，本文方法生成的描述展现出更高的具体性和准确性，同时在各项评价指标上均超过了目前的先进方法，有效验证了该方法的可行性。

关键词: 深度学习, 视频描述, 增强对象学习, 注意力网络

Abstract:

In video captioning tasks， one of the common problems is that the object caption is not specific enough， mainly because the model does not fully learn the information of the objects in the video. Meanwhile， videos contain abundant feature information， such as object information， motion information， and contextual information， making it a challenging task to enhance the model’s ability to learn key information when generating captions. To address the aforementioned problems， this paper proposes a method based on enhanced object learning and attention networks. Firstly， a new enhanced object learning module was designed to fully learn object information in videos， thereby achieving accurate caption of video content. Secondly， an attention network was constructed to dynamically adjust the weights of different types of information， thereby enhancing the model’s ability to learn key information when generating captions. In the experiments on the MSVD and MSR-VTT datasets， the caption generated by the method proposed in this paper showed a higher level of specificity and accuracy， and exceeded the current advanced methods in various evaluation indicators， effectively verifying the feasibility of the method.

Key words: deep learning, video captioning, enhanced object learning, attention network

中图分类号:

TP391

蔡晓东,龙顺宏,梁焜峻. 基于增强对象学习和注意力网络的视频描述方法[J]. 吉林大学学报(工学版), 2026, 56(2): 516-522.

Xiao-dong CAI,Shun-hong LONG,Kun-jun LIANG. Video captioning method based on enhanced object learning and attention networks[J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(2): 516-522.

图/表 6

图1

图2

图3

表1

表2

图4

参考文献 17

[1]	Zhang J, Peng Y. Video captioning with object-aware spatio-temporal correlation and aggregation[J]. IEEE Transactions on Image Processing, 2020, 29: 6209-6222.
[2]	Zanfir M, Marinoiu E, Sminchisescu C. Spatio-temporal attention models for grounded video captioning[C]∥Computer Vision-ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, 2017: 104-119.
[3]	Yang Z, Han Y, Wang Z. Catching the temporal regions-of-interest for video captioning[C]∥Proceedings of the 25th ACM International Conference on Multimedia, Mountain View, USA, 2017: 146-153.
[4]	Zhang W, Wang X E, Tang S, et al. Relational graph learning for grounded video description generation[C]∥Proceedings of the 28th ACM International Conference on Multimedia, Seattle, USA, 2020: 3807-3828.
[5]	Zhang Z, Shi Y, Yuan C, et al. Object relational graph with teacher-recommended learning for video captioning[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 13278-13288.
[6]	Kanani C S, Saha S, Bhattacharyya P. Global object proposals for improving multi-sentence video descriptions[C]∥International Joint Conference on Neural Network, Montreal, Canada, 2021: 1-7.
[7]	Parisotto E, Song F, Rae J, et al. Stabilizing transformers for reinforcement learning[C]∥International Conference on Machine Learning, Vienna, Austria, 2020: 7487-7498.
[8]	Ye H, Li G, Qi Y, et al. Hierarchical modular network for video captioning[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 17939-17948.
[9]	Lin K, Li L, Lin C C, et al. SwinBERT: end-to-end transformers with sparse attention for video captioning[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 17949-17958.
[10]	Gu X, Chen G, Wang Y, et al. Text with knowledge graph augmented transformer for video captioning[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, USA, 2023: 18941-18951.
[11]	Jing S, Zhang H, Zeng P, et al. Memory-based augmentation network for video captioning[J]. IEEE Transactions on Multimedia, 2023, 26: 2367-2379.
[12]	Shen Y, Gu X, Xu K, et al. Accurate and fast compressed video captioning[C]∥Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 2023: 15558-15567.
[13]	Wang J, Jiang W, Ma L, et al. Bidirectional attentive fusion with context gating for dense video captioning[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 7190-7198.
[14]	Wang B, Ma L, Zhang W, et al. Controllable video captioning with pos sequence guidance based on gated fusion network[C]∥Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 2019: 2641-2650.
[15]	Li L, Gao X, Deng J, et al. Long short-term relation transformer with global gating for video captioning[J]. IEEE Transactions on Image Processing, 2022, 31: 2726-2738.
[16]	Xu J, Yao T, Zhang Y, et al. Learning multimodal attention LSTM networks for video captioning[C]∥Proceedings of the 25th ACM International Conference on Multimedia, Mountain View, USA, 2017: 537-545.
[17]	Sun Z, Chen S, Zhong L. Visual-aware attention dual-stream decoder for video captioning[C]∥IEEE International Conference on Multimedia and Expo, Taipei, Taiwan, 2022: 1-6.

相关文章 15

[1]	姚宗伟,陈辰,高振云,靳鸿鹏,荣浩,李学飞,黄虹溥,毕秋实. 基于合成图像数据集的挖掘机关键点识别[J]. 吉林大学学报(工学版), 2026, 56(1): 76-85.
[2]	王琳虹,刘宇阳,刘子昱,鹿应佳,张宇恒,黄桂树. 基于YOLOv5的轻量化桥梁缺陷识别[J]. 吉林大学学报(工学版), 2025, 55(9): 2958-2968.
[3]	廉敬,张继保,刘冀钊,张家骏,董子龙. 基于文本引导的人脸图像修复[J]. 吉林大学学报(工学版), 2025, 55(8): 2732-2740.
[4]	刘元宁,王星喆,黄子彧,张家晨,刘震. 基于多模态数据融合的胃癌患者生存预测模型[J]. 吉林大学学报(工学版), 2025, 55(8): 2693-2702.
[5]	袁靖舒,李武,赵兴雨,袁满. 基于BERTGAT-Contrastive的语义匹配模型[J]. 吉林大学学报(工学版), 2025, 55(7): 2383-2392.
[6]	徐慧智,郝东升,徐小婷,蒋时森. 基于深度学习的高速公路小目标检测算法[J]. 吉林大学学报(工学版), 2025, 55(6): 2003-2014.
[7]	张汝波,常世淇,张天一. 基于深度学习的图像信息隐藏方法综述[J]. 吉林大学学报(工学版), 2025, 55(5): 1497-1515.
[8]	李健,刘欢,李艳秋,王海瑞,关路,廖昌义. 基于THGS算法优化ResNet-18模型的图像识别[J]. 吉林大学学报(工学版), 2025, 55(5): 1629-1637.
[9]	文斌,丁弈夫,杨超,沈艳军,李辉. 基于自选择架构网络的交通标志分类算法[J]. 吉林大学学报(工学版), 2025, 55(5): 1705-1713.
[10]	李振江,万利,周世睿,陶楚青,魏巍. 基于时空Transformer网络的隧道交通运行风险动态辨识方法[J]. 吉林大学学报(工学版), 2025, 55(4): 1336-1345.
[11]	赵孟雪,车翔玖,徐欢,刘全乐. 基于先验知识优化的医学图像候选区域生成方法[J]. 吉林大学学报(工学版), 2025, 55(2): 722-730.
[12]	金虎,申玉生,方勇,于丽,周佳媚. 基于深度学习SSD算法的公路隧道衬砌细小裂缝识别[J]. 吉林大学学报(工学版), 2025, 55(11): 3653-3659.
[13]	蔡晓东,黄业洋,董丽芳. 基于增强正例与层间负例的语义相似性模型[J]. 吉林大学学报(工学版), 2025, 55(11): 3705-3714.
[14]	姜来为,王策,杨宏宇. 基于深度学习的多目标跟踪研究进展综述[J]. 吉林大学学报(工学版), 2025, 55(11): 3429-3445.
[15]	王威,孙钰洁,王新. 频率和空间特征融合的轻量级多尺度遥感图像场景分类网络[J]. 吉林大学学报(工学版), 2025, 55(10): 3361-3371.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

方法	MSVD				MSR-VTT
方法	B4	M	C	R	B4	M	C	R
DMRM^［3］	51.1	33.6	74.8	—	—	—	—	—
ORG-TRL^［5］	54.3	36.4	95.2	73.9	43.6	28.8	50.9	62.1
POS-CG^［14］	52.5	34.1	88.7	71.3	42.0	28.2	48.7	61.6
LSRT^［15］	55.6	37.1	98.5	73.5	42.6	28.3	49.5	61.0
MA-LSTM^［16］	52.3	33.6	70.4	—	36.5	26.5	41.0	59.8
VADD^［17］	51.5	34.8	72.1	91.5	42.4	28.2	49.7	61.7
SwinBERT^［9］	58.2	41.3	120.6	77.5	41.9	29.9	53.8	62.1
TextKG^［10］	60.8	38.5	105.2	75.1	43.7	29.6	52.4	62.4
MAN^［11］	60.1	37.1	101.9	74.6	42.5	28.6	50.4	62.2
ViT/L14^［12］	60.1	41.4	121.5	78.2	44.4	30.3	57.2	63.4
HMN^［8］	59.2	37.7	104.0	75.1	43.5	29.0	51.5	62.7
EOLM-AN	61.5	39.0	106.7	77.5	44.2	29.4	52.1	63.6

方法	MSVD				MSR-VTT
方法	B4	M	C	R	B4	M	C	R
EOLM	61.3	38.1	105.3	77.2	44.1	29.1	51.7	63.4
AN	60.5	38.5	106.2	76.2	43.7	29.3	51.9	62.9
EOLM-AN	61.5	39.0	106.7	77.5	44.2	29.4	52.1	63.6