基于改进型DQN的无人机覆盖路径规划算法

doi:10.13229/j.cnki.jdxbgxb.20250237

Abstract

Abstract:

Aiming at the problems of low coverage， high path redundancy and slow convergence speed of traditional DQN in UAV coverage path planning， an improved DQN model based on multi-mechanism fusion was proposed. Firstly， the workspace of the UAV was modeled as a two-dimensional grid map. It was preprocessed and the reward function was designed. Secondly， based on the traditional DQN， three mechanisms of Double DQN， Dueling Network and Priority Experience Replay were gradually integrated， and a multi-scale convolutional neural network was built using techniques such as one-hot encoding to construct an improved DQN model. Finally， a visual simulation experiment was designed， and an ablation experiment was performed using evaluation indicators such as grid coverage. The experimental results show that the improved DQN model performs well in the coverage path planning task， and its performance is better than that of traditional DQN and other models. Compared with traditional DQN， its grid coverage rate is increased by 8.28%. Also， the grid repetition rate， path length， number of collisions with obstacles， and number of times exceeding boundaries are reduced by 28.44%， 45.14%， 77.31%， and 90.14%. The network convergence speed is significantly accelerated as well.

Key words: computer applications, deep reinforcement learning, coverage path planning, unmanned aerial vehicle, grid maps

CLC Number:

TP391

Xing-wang WANG,Jin-kuan LUO,Zi-yang ZHANG,Yu-pu CHI,Jin-you JIANG,Mei-ming YU. Algorithm of coverage path planning for unmanned aerial vehicle based on improved DQN[J].Journal of Jilin University(Engineering and Technology Edition), 2026, 56(9): 2444-2455.

Figures/Tables 14

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Table 1

Table 2

Experimental results of parameter sensitivity analysis"

改进型DQN模型相关参数	栅格覆盖率/%	栅格重复率/%	路径长度	碰撞障碍物次数	超出边界次数
ω=1e-4、γ=0.99、 $G ? i, j = G ? (i', j')$ 时奖励值+0.25	97.90	35.83	154.41	5.46	0.34
ω=1e-3	98.37	31.33	144.91	3.97	0.51
ω=1e-5	83.89	71.03	291.75	13.11	1.97
γ=0.8	96.91	43.36	173.43	3.40	2.44
γ=0.9	97.85	39.97	164.51	4.41	1.54
$G ? i, j = G ? (i', j')$ 时奖励值+0	92.41	53.94	230.00	15.79	2.01
$G ? i, j = G ? (i', j')$ 时奖励值+0.5	98.04	35.67	158.67	9.03	0.27

Table 2

Fig.8

Fig.9

Fig.10

Table 3

Table 4

References 20

[1]	Buchelt A, Adrowitzer A, Kieseberg P, et al. Exploring artificial intelligence for applications of drones in forest ecology and management[J]. Forest Ecology and Management, 2024, 551: No.121530.
[2]	Bakirci M. Smart city air quality management through leveraging drones for precision monitoring[J]. Sustainable Cities and Society, 2024, 106: No.105390.
[3]	Ibrahim Z T, He J. Slam technology on disaster response[J]. World Journal of Engineering and Technology, 2024, 12(3): 695-714.
[4]	Xing B, Wang X, Yang L, et al. An algorithm of complete coverage path planning for unmanned surface vehicle based on reinforcement learning[J]. Journal of Marine Science and Engineering, 2023, 11(3): No.645.
[5]	Kim J. Autonomous robot vacuum system composed of a cleaner robot and a dust storage robot[J]. Journal of the Franklin Institute, 2024, 361(11): No.106938.
[6]	Wan S, Chen Z, Dong J. An efficiency-based interactive dynamic technique with interval-valued hesitant fuzzy constraint cone for rescue route planning[J]. Expert Systems with Applications, 2023, 231: No.120648.
[7]	Dijkstra E W. A note on two problems in connexion with graphs[J]. Numerische Mathematik, 1959, 1(1): 269-271.
[8]	Hart P E, Nilsson N J, Raphael B. A formal basis for the heuristic determination of minimum cost paths[J]. IEEE transactions on Systems Science and Cybernetics, 1968, 4(2): 100-107.
[9]	Xu R, Xia Y, Chen P. A Multi-Objective Particle Swarm Optimization Algorithm for Drone Path Planning in Forest Firefighting[C]∥The 11th International Conference on Electrical and Electronics Engineering(ICEEE), Marmaris,Turkiye,2024:523-527.
[10]	唐颂, 吴建源. 基于改进遗传算法的协同航迹规划方法[J]. 电光与控制, 2024, 31(7): 8-12, 26.
	Tang Song, Wu Jian-yuan. A Cooperative Trajectory Planning Method Based on Improved Genetic Algorithm[J]. Electronics Optics & Control, 2024, 31(7): 8-12, 26.
[11]	Luo J, Shun H, Hao Y. A path planning method for comprehensive area coverage utilizing neuron activity[C]∥The 5th International Conference on Robotics, Intelligent Control and Artificial Intelligence(RICAI),Hangzhou,China, 2023: 379-383.
[12]	屈博琛. 基于深度学习自动微分的无人机路径规划[J]. 自动化与仪表, 2025, 40(2): 66-72.
	Qu Bo-chen. Drone path planning based on deep learning with automatic differentiation[J]. Automation & Instrumentation, 2025, 40(2): 66-72.
[13]	Mnih V, Kavukcuoglu K, Silver D, et al. Playing atari with deep reinforcement learning[J/OL].[2025-02-16]. .
[14]	van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double Q-learning[C]∥Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, Arizona, 2016, 30(1): 2094-2100.
[15]	Wang Z, Schaul T, Hessel M, et al. Dueling network architectures for deep reinforcement learning[C]∥International Conference on Machine Learning, New York, USA, 2016: 1995-2003.
[16]	Schaul T, Quan J, Antonoglou I, et al. Prioritized experience replay[J/OL]. [2025-01-22]. .
[17]	Watkins C J C H. Learning from delayed rewards[D]. Cambridge: King's College, University of Cambridge,1989.
[18]	Jaderberg M, Simonyan K, Zisserman A. Spatial transformer networks[J]. Advances in Neural Information Processing Systems, 2015, 28: 2017-2025.
[19]	Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018:7132-7141.
[20]	Brockman G, Cheung V, Pettersson L, et al. Openai gym[J/OL].[2025-01-26]. .

Related Articles 15

[1]	Tao JU,Wen-jin ZHANG,Yao YANG,Jiu-yuan HUO. A second order decision-making dynamic offloading method for vehicle edge computing tasks [J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(7): 2006-2019.
[2]	Cheng-jun TIAN,Yu YAN,Ren-wei CUI,Jin-tong ZHANG. Autonomous grasping algorithm of robotic arm based on deep reinforcement learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(3): 662-669.
[3]	Qing-lin AI,Yuan-xiao LIU,Jia-hao YANG. Small target swmantic segmentation method based MFF-STDC network in complex outdoor environments [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(8): 2681-2692.
[4]	Zi-hao SHEN,Yong-sheng GAO,Hui WANG,Pei-qian LIU,Kun LIU. Deep deterministic policy gradient caching method for privacy protection in Internet of Vehicles [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(5): 1638-1647.
[5]	Tao XU,Shuai-di KONG,Cai-hua LIU,Shi LI. Overview of heterogeneous confidential computing [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(3): 755-770.
[6]	Yao-ping ZENG,Yu-ting XIA,Shi-sen CHEN,Yue-qiang LIU,Wei-wei JIANG. Energy⁃efficient offloading strategy research for multi⁃UAV assisted communication [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(12): 4083-4092.
[7]	Wei-chao HU,Zhen-ming YANG,Peng-cheng YU,Yan-yan CHEN,She-qiang MA. Modeling interaction policy of autonomous vehicle and pedestrian based on deep reinforcement learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(10): 3180-3188.
[8]	Hai-yan HUANG,Hong-sheng ZHANG,Lin-lin LIANG,Chun-li WANG,Xue-jun ZHANG. Analysis of intelligent communication system with multi-UAVs based on SC/MRC [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(10): 3401-3409.
[9]	Yi TANG,Yang PAN,Ming GAO,Hong-chen YI,An-qi WEI. Multi spectral image matching algorithm of unmanned aerial vehicle based on affine invariant operator [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(7): 2080-2085.
[10]	Guang-he ZHU,Zhi-qiang ZHU,Yi-ping YUAN. Deep reinforcement learning optimization scheduling algorithm for continuous production line [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(7): 2086-2092.
[11]	Bin XIAN,Guang-yi WANG,Jia-ming CAI. Nonlinear robust control design for multi unmanned aerial vehicles suspended payload transportation system [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(6): 1788-1795.
[12]	Dian-wei WANG,Chi ZHANG,Jie FANG,Zhi-jie XU. UAV target tracking algorithm based on high resolution siamese network [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(5): 1426-1434.
[13]	Jing-peng GAO,Guo-xuan WANG,Lu GAO. LSTM⁃MADDPG multi⁃agent cooperative decision algorithm based on asynchronous collaborative update [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(3): 797-806.
[14]	Jian ZHANG,Qing-yang LI,Dan LI,Xia JIANG,Yan-hong LEI,Ya-ping JI. Merging guidance of exclusive lanes for connected and autonomous vehicles based on deep reinforcement learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(9): 2508-2518.
[15]	Ze-qiang ZHANG,Wei LIANG,Meng-ke XIE,Hong-bin ZHENG. Elite differential evolution algorithm for mixed⁃model two⁃side disassembly line balancing problem [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(5): 1297-1304.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

参数	含义	值
ω	网络学习率	1e-4
γ	折扣因子	0.99
?	见式（9）	1e-5
μ	见式（9）	0.6
σ	见式（11）	0.4
H	二维栅格地图高度	10
W	二维栅格地图宽度	10
frame_max	最大训练帧数	1 000 000
frame_start	训练开始帧数	10 000
\|D\|	回放经验池最大样本容量	100 000
\|K\|	可选动作的总数	4
Batch_size	每次训练选取的样本数	32
epsilon_max	最大贪婪系数	1.0
epsilon_min	最小贪婪系数	0.01
epsilon_decay	至最小贪婪系数的帧数	30 000
max_steps	每回合最大步数	300
agent_num	智能体数量	1
update_freq	网络参数更新间隔/帧	1 000
obstacles_per	障碍物栅格的占比/%	16

DDQN	Dueing network	PER	栅格覆盖率/%	栅格重复率/%	路径长度	碰撞障碍物次数	超出边界次数	训练时间	#Param /M	#FLOPs /G
?	?	?	89.62	64.27	281.45	24.06	3.45	3：22：04	3.72	1.11
?	?	?	93.92	61.83	273.66	23.78	0.85	4：14：59	3.72	1.11
?	?	?	93.72	56.13	233.79	12.72	3.65	3：56：05	7.13	1.22
?	?	?	94.47	51.18	215.99	11.69	2.62	3：26：30	3.72	1.11
?	?	?	94.82	50.40	219.92	19.37	0.71	4：31：45	7.13	1.22
?	?	?	96.26	44.76	187.66	10.76	0.77	4：07：36	7.13	1.22
?	?	?	95.94	48.76	201.16	10.05	3.34	3：38：55	3.72	1.11
?	?	?	97.90	35.83	154.41	5.46	0.34	4：36：02	7.13	1.22

模型	H，W	max_steps	栅格覆盖率/%	栅格重复率/%	路径长度	碰撞障碍物次数	超出边界次数
DQN	10	300	89.62	64.27	281.45	24.06	3.45
DQN+①	10	300	93.92	61.83	273.66	23.78	0.85
DQN+②	10	300	93.72	56.13	233.79	12.72	3.65
DQN+③	10	300	94.47	51.18	215.99	11.69	2.62
改进型DQN	10	300	97.90	35.83	154.41	5.46	0.34
DQN	12	350	84.68	63.46	349.70	20.36	26.14
DQN+①	12	350	86.14	64.84	348.76	13.07	43.30
DQN+②	12	350	85.16	61.47	345.89	24.45	31.46
DQN+③	12	350	89.80	58.61	329.08	18.76	12.15
改进型DQN	12	350	94.13	57.00	320.95	13.00	5.46
DQN	15	400	80.77	53.10	399.98	29.49	19.75
DQN+①	15	400	84.64	50.86	400.00	30.84	30.76
DQN+②	15	400	83.14	51.68	399.99	30.51	35.20
DQN+③	15	400	87.60	52.56	399.94	18.17	25.78
改进型DQN	15	400	92.70	49.66	391.99	14.29	4.96

Algorithm of coverage path planning for unmanned aerial vehicle based on improved DQN

RICH HTML

PDF (PC)