基于合成图像数据集的挖掘机关键点识别

doi:10.13229/j.cnki.jdxbgxb.20240677

摘要/Abstract

摘要：

本文提出了一种基于合成图像数据集的挖掘机关键点识别方法，通过虚拟模型和场景的随机自动化以及关键点坐标和遮挡信息判定，生成多样化的合成图像，并利用基于平面视觉的深度神经网络完成关键点识别，解决了传统大规模数据集采集困难的问题。试验结果显示：该方法提高了关键点识别精度，归一化误差和正确关键点百分比分别为0.005 6和97.64%。因此，本文方法能够满足监控挖掘机的作业安全和工作效率的实际应用需求，同时避免了高质量工程数据集采集时安全风险高、时间/经济成本高、工况覆盖面窄且标签准确率低等问题，有助于深度学习和大数据技术在挖掘机工作状态识别方面的应用部署。

关键词: 机械设计及理论, 挖掘机, 关键点识别, 深度学习, 合成图像

Abstract:

This paper proposes a method for excavator pose recognition using synthetic image datasets for model training. Initially， virtual models and scenarios are established. Programming is utilized to randomize the excavator pose， virtual camera position， and scene parameters. Subsequently， keypoint coordinates and occlusion information are computed to construct synthetic image datasets. Finally， excavator key points estimation is performed using a monocular camera. Experimental results demonstrate that training with synthetic image datasets improves model recognition accuracy， with a normalized error of 0.005 6 and a percentage of correct keypoint of 97.64%. Therefore， this method can meet the practical application needs of monitoring excavator operation safety and work efficiency. It also avoids issues such as high safety risks， high time/economic costs， narrow working condition coverage， and low label accuracy associated with high-quality engineering dataset collection. This contributes to the application and deployment of deep learning and big data technologies in excavator work state recognition.

Key words: mechanical design and theory, excavator, keypoints estimation, deep learning, synthetic datasets

中图分类号:

TP18

姚宗伟,陈辰,高振云,靳鸿鹏,荣浩,李学飞,黄虹溥,毕秋实. 基于合成图像数据集的挖掘机关键点识别[J]. 吉林大学学报(工学版), 2026, 56(1): 76-85.

Zong-wei YAO,Chen CHEN,Zhen-yun GAO,Hong-peng JIN,Hao RONG,Xue-fei LI,Hong-pu HUANG,Qiu-shi BI. Visual recognition of excavator keypoints based on synthetic image datasets[J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(1): 76-85.

图/表 12

图1

图2

图3

图4

图5

图6

图7

图8

图9

表1

表2

表3

参考文献 27

[1]	于向军, 槐元辉, 姚宗伟, 等. 工程车辆无人驾驶关键技术[J]. 吉林大学学报:工学版, 2021, 51(4): 1153-1168.
	Yu Xiang-jun, Huai Yuan-hui, Yao Zong-wei, et al. Key technologies in autonomous vehicle for engineering[J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(4): 1153-1168.
[2]	方成, 于盛鑫, 李永刚, 等. 基于深度学习的土木工程计算机视觉健康监测[J]. 同济大学学报:自然科学版, 2024, 52(2): 213-222.
	Fang Cheng, Yu Sheng-xin, Li Yong-gang, et al. Deep learning-based computer vision for health monitoring in civil engineering[J]. Journal of Tongji University(Natural Science), 2024, 52(2): 213-222.
[3]	Yao Z W, Zhao S C, Tan X D, et al. Real-time task-oriented continuous digging trajectory planning for excavator arms[J]. Automation in Construction, 2023, 152: No.104916.
[4]	Park J, Chen J D, Cho Y K. Self-corrective knowledge-based hybrid tracking system using bim and multimodal sensors[J]. Advanced Engineering Informatics, 2017, 32: 126-138.
[5]	Vahdatikhaki F, Hammad A, Siddiqui H. Optimization-based excavator pose estimation using real-time location systems[J]. Automation in Construction, 2015, 56: 76-92.
[6]	王太海, 陈建宏, 金俊. 基于挖掘机GNSS精确定位的开采姿态监测系统[J]. 黄金科学技术, 2016, 24(4): 101-106.
	Wang Tai-hai, Chen Jian-hong, Jin Jun. Mining attitude monitoring system based on GNSS precise positioning of excavator[J]. Gold Science and Technology, 2016, 24(4): 101-106.
[7]	Yang W J, Zhang X H, Ma H W, et al. Infrared leds-based pose estimation with underground camera model for boom-type roadheader in coal mining[J]. IEEE Access, 2019, 7: 33698-33712.
[8]	Assadzadeh A, Arashpour M, Brilakis I, et al. Vision-based excavator pose estimation using synthetically generated datasets with domain randomization[J]. Automation in Construction, 2022, 134: No.104089.
[9]	魏振忠, 冯广堃, 周丹雅, 等. 位姿视觉测量方法及应用综述[J]. 激光与光电子学进展, 2023, 60(3): 135-167.
	Wei Zhen-zhong, Feng Guang-kun, Zhou Dan-ya, et al. A review of position and orientation visual measurement methods and applications[J]. Laser & Optoelectronics Progress, 2023, 60(3): 135-167.
[10]	马伟, 宫乐, 冯浩, 等. 基于视觉的挖掘机工作装置位姿测量[J]. 机械设计与研究, 2018,34(5): 173-176, 182.
	Ma Wei, Gong Yue, Feng Hao, et al. Pose measurement of excavator device based on vision[J]. Machine Design and Research, 2018, 34(5): 173-176, 182.
[11]	王连明, 吴鑫. 基于姿态估计的物体3D运动参数测量方法[J]. 吉林大学学报:工学版, 2023, 53(7): 2099-2108.
	Wang Lian-ming, Wu Xin. Method for 3D motion parameter measurement based on pose estimation[J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(7): 2099-2108.
[12]	吴昊. 基于合成数据集的图像处理深度学习方法研究[D]. 兰州: 兰州大学信息科学与工程学院, 2021.
	Wu Hao. Research on deep learning methods for image processing based on synthetic datasets[D]. Lanzhou: School of Information Science and Engineering, Lanzhou University, 2021.
[13]	Kim J, Kim D, Lee S, et al. Hybrid DNN training using both synthetic and real construction images to overcome training data shortage[J]. Automation in Construction, 2023, 149: No.104771.
[14]	An X H, Zhou L, Liu Z G, et al. Dataset and benchmark for detecting moving objects in construction sites[J]. Automation in Construction, 2021, 122: No.103482.
[15]	Liang C J, Kamat V R, Menassa C M. Real-time construction site layout and equipment monitoring[C]∥Construction Research Congress, New Orleans,USA,2018: 64-74.
[16]	Zhang S B, Zhang L J. Construction site safety monitoring and excavator activity analysis system[J]. Construction Robotics, 2022, 6: 151-161.
[17]	郭晓新, 李佳慧, 张宝亮. 基于高分辨率网络的视杯和视盘的联合分割[J]. 吉林大学学报:工学版, 2023, 53(8): 2350-2357.
	Guo Xiao-xin, Li Jia-hui, Zhang Bao-liang. Joint segmentation of optic cup and disc based on high resolution network[J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(8): 2350-2357.
[18]	Tang J Y, Zhang X, Wong P K Y, et al. Method on pose estimation of excavators based on onboard depth camera[J]. IOP Conference Series: Earth and Environmental Science, 2022, 1101:No. 072005.
[19]	Lin T Y, Maire M, Belongie S, et al. Microsoft coco: common objects in context[J]. The 13th European Conference on Computer Vision, 2014, 8693: 740-755.
[20]	Ionescu C, Papava D, Olaru V, et al. Human3.6M: large scale datasets and predictive methods for 3d human sensing in natural environments[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2014, 36(7): 1325-1339.
[21]	Wen L Y, Kim D, Liu M Y, et al. 3D excavator pose estimation using projection-based pose optimization for contact-driven hazard monitoring [J]. Journal of Computing in Civil Engineering, 2023, 37: 1-15.
[22]	郑义桀, 罗健欣, 陈卫卫, 等. 基于Unity3D三维多视角虚拟数据集构建[J]. 计算机技术与发展, 2023, 33(5): 173-179.
	Zheng Yi-jie, Luo Jian-xin, Chen Wei-wei, et al. 3D multi-view virtual dataset construction based on Unity3D[J]. Computer Technology and Development, 2023, 33(5): 173-179.
[23]	Luo H, Wang M Z, Wong P K Y, et al. Full body pose estimation of construction equipment using computer vision and deep learning techniques[J]. Automation in Construction, 2020, 110: No.103016.
[24]	Torres C, Roberts D, Golparvar F M. Synthesizing pose sequences from 3D assets for vision-based activity analysis[J]. Journal of Computing in Civil Engineering, 2021, 35: No.04020052.
[25]	Tian Z H, Yu Y, Xu F, et al. Dynamic hazardous proximity zone design for excavator based on 3D mechanical arm pose estimation via computer vision[J]. Journal of Construction Engineering and Management, 2023, 149: 1-17.
[26]	Lou H T, Duan X H, Guo J M, et al. DC-YOLOv8: small-size object detection algorithm based on camera sensor[J]. Electronics, 2023, 12: No.2323.
[27]	Mahmood B, Han S, Seo J. Implementation experiments on convolutional neural network training using synthetic images for 3D pose estimation of an excavator on real images[J]. Automation in Construction, 2022, 133: No.103996.

相关文章 15

[1]	王琳虹,刘宇阳,刘子昱,鹿应佳,张宇恒,黄桂树. 基于YOLOv5的轻量化桥梁缺陷识别[J]. 吉林大学学报(工学版), 2025, 55(9): 2958-2968.
[2]	刘志峰,李伟宁,齐宝宝,陈传海,侍威,张兆晶,谭晓庆,邬昌军. 行星滚柱丝杠滚滑运动机理分析及试验[J]. 吉林大学学报(工学版), 2025, 55(9): 2892-2901.
[3]	刘元宁,王星喆,黄子彧,张家晨,刘震. 基于多模态数据融合的胃癌患者生存预测模型[J]. 吉林大学学报(工学版), 2025, 55(8): 2693-2702.
[4]	廉敬,张继保,刘冀钊,张家骏,董子龙. 基于文本引导的人脸图像修复[J]. 吉林大学学报(工学版), 2025, 55(8): 2732-2740.
[5]	袁靖舒,李武,赵兴雨,袁满. 基于BERTGAT-Contrastive的语义匹配模型[J]. 吉林大学学报(工学版), 2025, 55(7): 2383-2392.
[6]	徐慧智,郝东升,徐小婷,蒋时森. 基于深度学习的高速公路小目标检测算法[J]. 吉林大学学报(工学版), 2025, 55(6): 2003-2014.
[7]	张汝波,常世淇,张天一. 基于深度学习的图像信息隐藏方法综述[J]. 吉林大学学报(工学版), 2025, 55(5): 1497-1515.
[8]	李健,刘欢,李艳秋,王海瑞,关路,廖昌义. 基于THGS算法优化ResNet-18模型的图像识别[J]. 吉林大学学报(工学版), 2025, 55(5): 1629-1637.
[9]	文斌,丁弈夫,杨超,沈艳军,李辉. 基于自选择架构网络的交通标志分类算法[J]. 吉林大学学报(工学版), 2025, 55(5): 1705-1713.
[10]	李振江,万利,周世睿,陶楚青,魏巍. 基于时空Transformer网络的隧道交通运行风险动态辨识方法[J]. 吉林大学学报(工学版), 2025, 55(4): 1336-1345.
[11]	付学中,何厚冰,刘旭东,李静珍. 非正交偏置变位一体化的斜齿面齿轮齿宽设计[J]. 吉林大学学报(工学版), 2025, 55(4): 1207-1214.
[12]	赵孟雪,车翔玖,徐欢,刘全乐. 基于先验知识优化的医学图像候选区域生成方法[J]. 吉林大学学报(工学版), 2025, 55(2): 722-730.
[13]	金虎,申玉生,方勇,于丽,周佳媚. 基于深度学习SSD算法的公路隧道衬砌细小裂缝识别[J]. 吉林大学学报(工学版), 2025, 55(11): 3653-3659.
[14]	陈一馨,陈再续,刘永生,杨帅,郭浩杰,贾晋三. 基于改进深度代理模型的挖掘机铲斗结构优化设计[J]. 吉林大学学报(工学版), 2025, 55(11): 3485-3497.
[15]	姜来为,王策,杨宏宇. 基于深度学习的多目标跟踪研究进展综述[J]. 吉林大学学报(工学版), 2025, 55(11): 3429-3445.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

数据集	Point_1	Point_2	Point_3	Point_4	Point_5	Point_6	Point_7	平均
Dataset #1	99.95	99.05	98.57	82.38	99.99	94.24	99.76	96.28
Dataset #2	99.99	99.99	99.99	83.76	99.99	99.76	99.95	97.64
Tian等^［1］	98.75	95.00	88.57	80.10	—	—	—	90.61
Mahmood等^［2］	96.18	96.94	94.65	92.78	—	—	—	95.14

数据集	Point_1	Point_2	Point_3	Point_4	Point_5	Point_6	Point_7	平均
Dataset #1	0.003 8	0.004 3	0.006 1	0.012 8	0.002 2	0.006 1	0.005 9	0.005 9
Dataset #2	0.003 4	0.004 2	0.006 0	0.013 1	0.003 1	0.004 8	0.004 9	0.005 6
Assadzadeh等^［3］	0.028 7	0.020 8	0.031 1	0.099 2	—	—	—	0.045 0
Wen等^［4］	0.039 9	0.028 2	0.021 9	0.029 0	—	—	—	0.029 8

数据集	PCK/%
数据集	挖掘	回转	卸料
Dataset #1	59.19	89.63	99.99
Dataset #2	67.22	89.05	95.30