一种二阶决策的车联网边缘计算任务动态卸载方法

doi:10.13229/j.cnki.jdxbgxb.20241283

摘要/Abstract

摘要：

为解决车联网边缘计算中处理离散-连续混合决策问题时边缘计算系统计算性能不高的问题，基于深度强化学习提出了一种分布式的二阶决策车联网边缘计算任务动态卸载方法。通过将任务处理过程分为卸载决策和资源分配两个子问题，基于D3DQN网络和TD3网络设计了可同时处理离散和连续动作域的算法框架；同时，设计了动作空间搜索优化和动态优先级更新机制，以进一步提升对动作空间的有效搜索，提高算法性能；最终，在以上算法框架和优化机制的基础上，设计实现了二阶决策车联网边缘计算任务动态卸载算法。该算法可有效解决车联网边缘计算场景中的离散-连续混合决策问题，实现边缘计算系统资源的有效利用，保证车辆能根据当前网络状态和任务大小，以最小的时延和能耗完成计算任务卸载。仿真结果表明，与基准卸载方法相比，本文方法具有更快的收敛性和更低的时延能耗，可以较好地利用边缘系统计算资源为车载边缘任务请求提供计算服务，进一步提升用户服务质量。

关键词: 车联网, 移动边缘计算, 深度强化学习, 动态任务卸载, 资源分配

Abstract:

In response to the issue of low computational performance of edge computing systems when dealing with discrete-continuous hybrid decision-making problems in vehicular network edge computing scenarios， a distributed multi-stage decision-making strategy for dynamic offloading of vehicular network edge computing tasks based on deep reinforcement learning is proposed. By dividing the task processing procedure into two sub-problems： offloading decisions and resource allocation， an algorithm framework capable of handling both discrete and continuous action domains was designed based on the D3DQN network and TD3 network. In order to further enhance effective search in the action space and improve algorithm performance， an optimization and dynamic priority update mechanism for search in the action space was designed. Finally， based on the above algorithm framework and optimization mechanisms， a multi-stage decision-making algorithm for dynamic offloading of vehicle network edge computing tasks was developed and implemented. This algorithm can effectively solve discrete-continuous hybrid decision-making problems in vehicle network edge computing scenarios， utilize edge computing system resources， and ensure that vehicles can complete the offloading of computing tasks with minimal delay and energy consumption， according to the current network status and task size. Simulation results show that， compared with baseline offloading methods， the proposed method has faster convergence and lower latency energy consumption. It can make better use of the computing resources of the edge system to provide computing services for vehicle edge task requests， thus further enhancing the quality of service for users.

Key words: internet of vehicles, mobile edge computing, deep reinforcement learning, dynamic task offloading, resource allocation

中图分类号:

TP311

巨涛,张文金,杨垚,火久元. 一种二阶决策的车联网边缘计算任务动态卸载方法[J]. 吉林大学学报(工学版), 2026, 56(7): 2006-2019.

Tao JU,Wen-jin ZHANG,Yao YANG,Jiu-yuan HUO. A second order decision-making dynamic offloading method for vehicle edge computing tasks[J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(7): 2006-2019.

图/表 17

图1

图2

图3

图4

图5

表1

实验参数"

参数	数值
车辆数量	20
基站覆盖范围（直径）/m	1 200
高斯白噪声N₀/dBm	-110
参考信道功率增益β₀/dB	-30
车辆最大计算资源f_max/ GHz	IPC41.2*3
任务复杂度C/（cycles·bit^-1）	［50，1 250］
任务最大时延 $T ?$ /s	｛0.75，1.5｝
路径损耗指数θ	4
车辆行驶速度/（km·h^-1）	50
单RSU覆盖范围（直径）/m	400
无线通信带宽B/ MHz	5
传输功率p/ mW	100
VEC最大计算资源F_max/GHz	IPC321.2*4
任务大小D/kB	［100，1 000］
单位CPU周期能耗κ	10^-28
批处理尺寸batchsize	64

表1

图6

图7

图8

图9

图10

图11

图12

图13

图14

图15

图16

参考文献 33

[1]	Sharma S, Kaushik B. A survey on internet of vehicles: applications, security issues & solutions[J]. Vehicular Communications, 2019, 20:100182.
[2]	况博裕, 李雨泽, 顾芳铭,等. 车联网安全研究综述: 威胁、对策与未来展望[J]. 计算机研究与发展, 2023, 60(10): 2304-2321.
	Kuang Bo-yu, Li Yu-ze, Gu Fang-ming, et al. Review of internet of vehicle security research: threats, countermeasures, and future prospects[J]. Journal of Computer Research and Development, 2023, 60(10): 2304-2321.
[3]	Boban M, Kousaridas A, Manolakis K, et al. Connected roads of the future: use cases, requirements, and design considerations for vehicle-to-everything communications[J]. IEEE Vehicular Technology Magazine, 2018, 13(3): 110-123.
[4]	Yang D G, Jiang K, Zhao D, et al. Intelligent and connected vehicles: current status and future perspectives[J]. Science China Technological Sciences, 2018, 61: 1446-1471.
[5]	Zhang W Y, Zhang Z J, chao H C. Cooperative fog computing for dealing with big data in the internet of vehicles: architecture and hierarchical resource management[J]. IEEE Communications Magazine, 2017, 55(12): 60-67.
[6]	Jaehwan L, Woongsoo N. A survey on vehicular edge computing architectures[C]∥2022 13th International Conference on Information and Communication Technology Convergence(ICTC), Jeju, Island, Korea, 2022: 2198-2200.
[7]	Mao Y Y, You C S, Zhang J, et al. A survey on mobile edge computing: the communication perspective[J]. IEEE Communications Surveys & Tutorials, 2017, 19(4): 2322-2358.
[8]	Tang M, Wong V W S. Deep reinforcement learning for task offloading in mobile edge computing systems[J]. IEEE Transactions on Mobile Computing, 2022, 21(6): 1985-1997.
[9]	Zhao N, Liang Y C, Niyato D, et al. Deep reinforcement Learning for user association and resource allocation in heterogeneous cellular networks[J]. IEEE Transactions on Wireless Communications, 2019, 18(11): 5141-5152.
[10]	Dai Y Y, Xu D, Zhang K, et al. Deep reinforcement learning for edge computing and resource allocation in 5G beyond[C]∥2019 IEEE 19th International Conference on Communication Technology, Xi'an, China, 2019: 866-870.
[11]	Raeisi-varzaneh M, Dakkak O, Habbal A, et al. Resource scheduling in edge computing: architecture, taxonomy, open issues and future research directions[J]. IEEE Access, 2023, 11: 25329-25350.
[12]	Feng J, Yu F R, Pei Q Q, et al. Joint optimization of radio and computational resources allocation in blockchain-enabled mobile edge computing systems[J]. IEEE Transactions on Wireless Communications, 2020, 19(6): 4321-4334.
[13]	李智勇, 王琦, 陈一凡, 等. 车辆边缘计算环境下任务卸载研究综述[J]. 计算机学报, 2021, 44(5): 963-982.
	Li Zhi-yong, Wang Qi, Chen Yi-fan, et al. A survey on task offloading research in vehicular edge computing[J]. Chinese Journal of Computers, 2021, 44(5): 963-982.
[14]	许斌, 赵云凯, 朱剑鸣, 等. 移动边缘计算不确定性任务持续卸载及资源分配方法[J]. 软件学报, 2023, 35(3): 1466-1484.
	Xu Bin, Zhao Yun-kai, Zhu Jian-ming, et al. Continuous offloading and resource allocation method of uncertain tasks in mobile edge computing[J]. Journal of Software, 2023, 35(3): 1466-1484.
[15]	Liu G Z, Dai F, Huang B, et al. A collaborative computation and dependency-aware task offloading method for vehicular edge computing: a reinforcement learning approach[J]. Journal of Cloud Computing, 2022, 11(1): 1-15.
[16]	孙彦景, 余政达, 陈瑞瑞, 等. 车联网中基于深度强化学习的高可靠资源分配算法[J]. 重庆邮电大学学报: 自然科学版, 2023, 35(4): 706-714.
	Sun Yan-jing, Yu Zheng-da, Chen Rui-rui, et al. Deep reinforcement learning based high reliability resource allocation algorithm for internet of vehicles[J]. Journal of Chongqing University of Posts and Telecommunications(Natural Science Edition), 2023, 35(4): 706-714.
[17]	Hao H, Ding W, Zhang W. Time-continuous computing offloading algorithm with user fairness guarantee[J]. Journal of Network and Computer Applications, 2024, 223: 103826.
[18]	Yan J, Bi S Z, Zhang Y J A. Offloading and resource allocation with general task graph in mobile edge computing: a deep reinforcement learning approach[J]. IEEE Transactions on Wireless Communications, 2020, 19(8): 5404-5419.
[19]	Zhao J H, Li Q P, Gong Y, et al. Computation offloading and resource allocation for cloud assisted mobile edge computing in vehicular networks[J]. IEEE Transactions on Vehicular Technology, 2019, 68(8): 7944-7956.
[20]	Han H, Wu D G, Zhou F H, et al. Dynamic task offloading in MEC-enabled IoT networks: a hybrid DDPG-D3QN approach[C]∥2021 IEEE Global Communications Conference(GLOBECOM). Madrid, Spain, 2021: 1-6.
[21]	Guan T, Yao Y S, Chao Y, et al. Energy-efficient computing offloading policy for MEC-assisted power sensor network: a D3QN approach[C]∥2023 International Conference on Cyber-Physical Social Intelligence(ICCSI), Xi'an, China, 2023: 180-185.
[22]	Wang Z Y, Schaul T, Hessel M, et al. Dueling Network Architectures for Deep reinforcement learning[C]∥JMLR.org. Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48, New York, NY, USA, 2016: 1995-2003.
[23]	Hasselt H V, Guez A, Silver D. Deep reinforcement learning with double Q-learning[C]∥AAAI'16. Proceeding of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, Arizona, 2016: 2094-2100.
[24]	巨涛, 王志强, 刘帅, 等. D3DQN-CAA: 一种基于DRL的自适应边缘计算任务调度方法[J].湖南大学学报: 自然科学版, 2024, 51(6): 73-85.
	Ju Tao, Wang Zhi-qiang, Liu Shuai, et al. D3DQN-CAA:a DRL-based adaptive edge computing task scheduling method[J]. Journal of Hunan University(Natural Sciences), 2024, 51(6): 73-85.
[25]	张红霞, 吕智豪, 席诗语, 等. 面向绿色计算的车辆协同任务卸载方法[J]. 电子与信息学报, 2024, 48(1): 175-183.
	Zhang Hong-Xia, Lv Zhi-hao, Xi Shi-yu, et al. A method for offloading vehicle collaborative tasks for green computing[J]. Journal of Electronics & Information Technology, 2024, 48(1): 175-183.
[26]	Kiran B R, Sobh I, Talpaert V, et al. Deep reinforcement learning for autonomous driving: A survey[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(6): 4909-4926.
[27]	Zhou D R, Li L H, Gu Q Q. Neural contextual bandits with ucb-based exploration[C]∥International Conference on Machine Learning. 2020: 11492-11502.
[28]	Zhong J W, Chen X, Jiao L B. A fair energy-efficient offloading scheme for heterogeneous fog-enabled IoT using D3QN[C]∥2023 25th International Conference on Communication Software and Networks(ICCSN), IEEE, 2023: 47-52.
[29]	Min M H, Xiao L, Chen Y, et.al. Learning-based computation offloading for IoT devices with energy harvesting[J]. IEEE Transactions on Vehicular Technology, 2019, 68(2): 1930-1941.
[30]	Geng L W, Zhao H B, Wang J Y, et al. Deep-reinforcement-learning-based distributed computation offloading in vehicular edge computing networks[J]. IEEE Internet of Things Journal, 2023, 10(14): 12416-12433.
[31]	郭晓东, 郝思达, 王丽芳. 基于深度强化学习的车辆边缘计算任务卸载方法[J]. 计算机应用技术, 2023, 40(9): 2803-2807, 2814.
	Guo Xiao-dong, Hao Si-da, Wang Li-fang. Task offloading method based on deep reinforcement learning for vehicular edge computing[J]. Application Research of Computers, 2023, 40(9): 2803-2807, 2814.
[32]	邝祝芳, 陈清林, 李林峰, 等. 基于深度强化学习的多用户边缘计算任务卸载调度与资源分配算法[J]. 计算机学报, 2022, 45(4): 812-824.
	Kuang Zhu-fang, Chen Qing-li, Li Li-feng, et al. Multi-user edge computing task offloading scheduling and resource allocation based on deep reinforcement learning[J]. Chinese Journal of Computers, 2022, 45(4): 812-824.
[33]	许小龙, 方子介, 齐连永, 等. 车联网边缘计算环境下基于深度强化学习的分布式服务卸载方法[J]. 计算机学报, 2021, 44(12): 2382-2405.
	Xu Xiao-long, Fang Zi-jie, Qi Lian-yong, et al. A deep reinforcement learning-based distributed service offloading method for edge computing empowered internet of vehicles[J]. Chinese Journal of Computers, 2021, 44(12): 2382-2405.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed