基于深度强化学习的自动驾驶车辆专用道汇入引导

doi:10.13229/j.cnki.jdxbgxb.20220106

Abstract

Abstract:

Exclusive lanes for connected and autonomous vehicles（CAVs） will emerge in order to ensure the safety and efficiency requirements in the process of traffic flow mixed with human-driving vehicles and CAVs. When the inner lane of the expressway is set as the exclusive lane for CAVs， it has important theoretical significance and practical value to study the strategy of guiding CAVs to merge from the ordinary lane to the exclusive lane. Firstly， the entrance area of exclusive lane was designed and vehicle control rules were proposed. Secondly， with the goal of making more CAVs change lanes to the exclusive lane， the strategy of selecting lane-changing signal actions was proposed based on deep reinforcement learning. Finally， the numerical simulation was carried out with Python language compilation. The results show that the proposed algorithm can converge very quickly under 9 scenarios constructed by different factors， such as the CAV penetration rates and the proportion of CAVs arriving at the exclusive lane； it can effectively guide CAVs to merge into exclusive lanes and ensure traffic efficiency； congestion in the second lane can be significantly reduced compared to the unsignalized control when the penetration rate changes from 20% to 40%； the proportion of CAVs changing to the exclusive lane is significantly higher under the two exclusive lane entrances scenario than that under the one entrance scenario. It shows that the proposed strategy has good applicability and can provide reference for engineering construction.

Key words: engineering of transportation system, merging guidance strategy, deep reinforcement learning, expressway exclusive lane, signal control, connected and autonomous vehicle

CLC Number:

U491

Jian ZHANG,Qing-yang LI,Dan LI,Xia JIANG,Yan-hong LEI,Ya-ping JI. Merging guidance of exclusive lanes for connected and autonomous vehicles based on deep reinforcement learning[J].Journal of Jilin University(Engineering and Technology Edition), 2023, 53(9): 2508-2518.

Figures/Tables 12

Fig.1

Fig.2

Fig.3

Table 1

Vehicle control method parameters"

参数	$l c e l l$	$v m a x$	$τ$	$p l e f t$	$p r i g h t$	$p$	$p 0$	$p a 1$	$p a 2$
取值	1.5	30	1	0.4	0.3	0.04	0.425	0.2	0.052

Table 1

Table 2

Merging guiding strategy parameters"

参数	$l l e n$	$l w i d$	$ε i n i t i a l$	$ε f i n a l$	$γ$	$n p o o l$	$n b a t c h$	$φ$
取值	4.5	1.8	0.5	0.01	0.9	5000	64	0.045

Table 2

Fig.4

Fig.5

Fig.6

Table 3

Capacity under different scenarios"

方法		$P$ =20%			$P$ =30%			$P$ =40%
方法		$ρ$ =20%	$ρ$ =40%	$ρ$ =60%	$ρ$ =20%	$ρ$ =40%	$ρ$ =60%	$ρ$ =20%	$ρ$ =40%	$ρ$ =60%
	本文模型	3301	3426	3303	3293	3282	3377	3335	3301	3426
	无信号控制	3323	3425	3303	2736	3273	3377	3339	3323	3425

Table 3

Fig.7

Fig.8

Fig.9

References 31

1	秦严严, 王昊, 王炜. 智能网联环境下的混合交通流LWR模型[J]. 中国公路学报, 2018, 31(11): 147-156.
	Qin Yan-yan, Wang Hao, Wang Wei. LWR model for mixed traffic flow in connected and autonomous vehicular environments[J]. China Journal of Highway and Transport, 2018, 31(11): 147-156.
2	Becker F, Axhausen K W. Literature review on surveys investigating the acceptance of automated vehicles[J]. Transportation, 2017, 44(6): 1293-1306.
3	Elliott D, Keen W, Miao L. Recent advances in connected and automated vehicles[J]. Journal of Traffic and Transportation Engineering (English Edition), 2019, 6(2): 109-131.
4	杜豫川, 刘成龙, 吴荻非, 等. 新一代智慧高速公路系统架构设计[J]. 中国公路学报, 2022, 35(4): 203-214.
	Du Yu-chuan, Liu Cheng-long, Wu Di-fei, et al. Framework of the new generation of smart highway[J]. China Journal of Highway and Transport, 2022, 35(4): 203-214.
5	Shladover S E. Connected and automated vehicle systems: introduction and overview[J]. Journal of Intelligent Transportation Systems, 2017, 22(3): 190-199.
6	冉斌, 谭华春, 张健, 等. 智能网联交通技术发展现状及趋势[J]. 汽车安全与节能学报, 2018, 9(2): 119-130.
	Ran Bin, Tan Hua-chun, Zhang Jian, et al. Development status and trend of connected automated vehicle highway system[J]. Journal of Automotive Safety and Energy, 2018, 9(2): 119-130.
7	Ma K, Wang H. Influence of exclusive lanes for connected and autonomous vehicles on freeway traffic flow[J]. IEEE Access, 2019, 7: 50168-50178.
8	Hua X D, Yu W J, Wang W. Influence of lane policies on freeway traffic mixed with manual and connected and autonomous vehicles[J]. Journal of Advanced Transportation, 2020, 2020: No.3968625.
9	Ghiasi A, Hussain O, Qian Z. A mixed traffic capacity analysis and lane management model for connected automated vehicles: a markov chain method[J]. Transportation Research Part B, 2017, 106: 266-292.
10	Chen D J, Ahn S, Chitturi M. Towards vehicle automation: roadway capacity formulation for traffic mixed with regular and automated vehicles[J]. Transportation Research Part B, 2017, 100: 196-221.
11	Ye L, Yamamoto T. Impact of dedicated lanes for connected and autonomous vehicle on traffic flow throughput[J]. Physica A, 2018, 512: 588-597.
12	Ye L, Yamamoto T. Modeling connected and autonomous vehicles in heterogeneous traffic flow[J]. Physica A, 2018, 490: 269-277.
13	Masher D P, Ross D W, Wong P J, et al. Guidelines for design and operation of ramp control systems[R]. Menlo Park: CA United States: Stanford Research Institute,1975.
14	Smaragdis E, Papageogiou M. Series of new local ramp metering strategies[J]. Transportation Research Record, 2003, 1856: 74-86.
15	Bellemans T, Schutter B D, Moor B D. Model predictive control for ramp metering of motorway traffic: a case study[J]. Control Engineering Practice, 2006, 14(7): 757-767.
16	陈德望, 王飞跃, 陈龙. 基于模糊神经网络的城市高速公路入口匝道控制算法[J]. 交通运输工程学报, 2003(2): 100-105.
	Chen De-wang, Wang Fei-yue, Chen Long. Freeway ramp control algorithm based on neurofuzzy networks[J]. Journal of Traffic and Transportation Engineering, 2003(2): 100-105.
17	曾筠程, 邵敏华, 孙立军, 等. 基于有向图卷积神经网络的交通预测与拥堵管控[J]. 中国公路学报, 2021, 34(12): 239-248.
	Zeng Yun-cheng, Shao Min-hua, Sun Li-jun, et al. Traffic prediction and congestion control based on directed graph convolution neural network[J]. China Journal of Highway and Transport, 2021, 34(12): 239-248.
18	Zhang J, Jiang X, Liu Z, et al. A study on autonomous intersection management: planning-based strategy improved by convolutional neural network[J]. KSCE Journal of Civil Engineering, 2021, 25(10): 3995-4004.
19	Davarynejad M, Hegyi A, Vrancken J, et al. Motorway ramp-metering control with queuing consideration using Q-learning[C]∥2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), Washington, DC, USA, 2011: 1652-1658.
20	郑思. 面向快速道路远距离瓶颈的深度强化学习交通流控制策略研究[D]. 南京:东南大学交通学院, 2021.
	Zheng Si. Research on deep reinforcement learning-based active traffic flow control strategies at distant downstream bottlenecks of expressway[D]. Nanjing: School of Transportation, Southeast University, 2021.
21	Belletti F, Haziza D, Gomes G. Expert level control of ramp metering based on multi-task deep reinforcement learning[J]. Transactions on Intelligent Transportation Systems, 2017, 19(4): 1198-1207.
22	葛家丽. 基于车辆簇的高速公路路侧单元部署研究[D]. 济南: 山东科技大学交通学院, 2020.
	Ge Jia-li. Research on freeway road side unit deployment based on vehicle clusters[D]. Jinan: School of Transportation, Shandong University of Science and Technology, 2020.
23	陈瑜. 高速公路作业区安全分析及交通组织管理方法研究[D]. 哈尔滨: 哈尔滨工业大学交通科学与工程学院, 2006.
	Chen Yu. The safety analysis and organization & management method study of freeway work zone[D]. Harbin: School of Transportation Science & Engineering, Harbin Institute of Technology, 2006.
24	石茂清. 道路交通安全设施设计研究[D]. 成都: 西南交通大学交通运输与物流学院, 2005.
	Shi Mao-qing. Study on design of traffic safety facilities[D]. Chengdu: School of Transportation and Logistics, Southwest Jiaotong University, 2005.
25	孙玲, 张静, 周瀛, 等. 车路协同环境下自动驾驶专用车道入口区域设计[J]. 公路交通科技, 2020, 37(): 122-129.
	Sun Ling, Zhang Jing, Zhou Ying, et al. Design of entrance area of automatic driving special lane in vehicle-infrastructure collaborative environment[J]. Journal of Highway and Transportation Research and Development, 2020, 37(Sup.1): 122-129.
26	Milanés V, Shladover S E. Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data[J]. Transportation Research Part C: Emerging Technologies, 2014, 48: 285-300.
27	Xue Y, Wang X, Cen B L, et al. Study on fuel consumption in the Kerner-Klenov-Wolf three-phase cellular automaton traffic flow model[J]. Nonlinear Dynamics, 2020, 102(1): 1-10.
28	Wang C H, Hwang M C. Value-based deep reinforcement learning for adaptive isolated intersection signal control[J]. IET Intelligent Transport Systems, 2018, 12(9): 1005-1010.
29	Schaul T, Quan J, Antonoglou I, et al. Prioritized experience replay[C]∥Proceeding of the 4th International Conference on Learning Representations, San Juan, Puerto Rico, 2016: 322-355.
30	武毅. 基于三相交通流理论的元胞自动机模型研究[D]. 长春: 吉林大学交通学院, 2018.
	Wu Yi. Research on cellular automated traffic flow model for three-phase theory[D]. Changchun: College of Transportation, Jilin University, 2018.
31	舒凌洲, 吴佳, 王晨. 基于深度强化学习的城市交通信号控制算法[J]. 计算机应用, 2019, 39(5): 1495-1499.
	Shu Ling-zhou, Wu Jia, Wang Chen. Urban traffic signal control based on deep reinforcement learning[J]. Journal of Computer Applications, 2019, 39(5): 1495-1499.

Related Articles 15

[1]	Yan-tao TIAN,Yan-shi JI,Huan CHANG,Bo XIE. Deep reinforcement learning augmented decision⁃making model for intelligent driving vehicles [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(3): 682-692.
[2]	Wei-chao ZHUANG,Hao-nan Ding,Hao-xuan DONG,Guo-dong YIN,Xi WANG,Chao-bin ZHOU,Li-wei XU. Learning based eco⁃driving strategy of connected electric vehicle at signalized intersection [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(1): 82-93.
[3]	Yong LIU,Lei XU,Chu-han ZHANG. Deep reinforcement learning model for text games [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 666-674.
[4]	Zhong-li WANG,Hao WANG,Yan SHEN,Bai-gen CAI. A driving decision⁃making approach based on multi⁃sensing and multi⁃constraints reward function [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(11): 2718-2727.
[5]	Dong-bo LIU,Li-xiao SHEN,Lei-lei DAI,Jian LU. Traffic signal control method at isolated intersections based on multi-target radar data [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(10): 2456-2465.
[6]	Ya-hui ZHAO,Fei-yang YANG,Zhen-guo ZHANG,Rong-yi CUI. Korean text structure discovery based on reinforcement learning and attention mechanism [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(4): 1387-1395.
[7]	Kai LU,Wei WU,Guan-rong LIN,Xin TIAN,Jian-min XU. Combination forecasting model for number of assembling passengers at transportation terminal based on KNN regression algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(4): 1241-1250.
[8]	Shuai LYU,Jing LIU. Stochastic local search heuristic method based on deep reinforcement learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(4): 1420-1426.
[9]	Jian ZHANG,Kun-run WU,Min YANG,Bin RAN. Double⁃ring adaptive control model of intersection during intelligent and connected environment [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(2): 541-548.
[10]	Yi-ming BIE,Kai JIANG,Ru-ru TANG,Lin-hong WANG,Xin-yu XIONG. Time of interval partition for traffic control at isolated intersection considering impacts of plan transition [J]. Journal of Jilin University(Engineering and Technology Edition), 2019, 49(6): 1844-1851.
[11]	Shun YANG,Yuan⁃de JIANG,Jian WU,Hai⁃zhen LIU. Autonomous driving policy learning based on deep reinforcement learning and multi⁃type sensor data [J]. Journal of Jilin University(Engineering and Technology Edition), 2019, 49(4): 1026-1033.
[12]	CHEN Song, LI Xian-sheng, REN Yuan-yuan. Adaptive signal control method for intersection with hook-turn buses [J]. 吉林大学学报(工学版), 2018, 48(2): 423-429.
[13]	ZHAO Shu-zhi, LIANG Shi-dong, MA Ming-hui, LIU Hua-sheng, ZHU Yong-gang. Real-time queue length estimation at signalized intersection [J]. 吉林大学学报(工学版), 2016, 46(1): 85-91.
[14]	XU Hong-feng, ZHANG Kun, YAO Rong-han. Control strategy of full metering signalization at roundabout [J]. 吉林大学学报(工学版), 2016, 46(1): 76-84.
[15]	XU Hong-feng, GENG Xian-cai, HE Long. Signal timing plan for fully signalized four-leg roundabouts with single-approach-entering operation [J]. 吉林大学学报(工学版), 2014, 44(4): 953-962.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 10

[1]	LI Shoutao, LI Yuanchun. Autonomous Mobile Robot Control Algorithm Based on Hierarchical Fuzzy Behaviors in Unknown Environments[J]. 吉林大学学报(工学版), 2005, 35(04): 391 -397 .
[2]	Liu Qing-min，Wang Long-shan，Chen Xiang-wei，Li Guo-fa. Ball nut detection by machine vision[J]. 吉林大学学报(工学版), 2006, 36(04): 534 -538 .
[3]	Li Hong-ying; Shi Wei-guang;Gan Shu-cai. Electromagnetic properties and microwave absorbing property of Z type hexaferrite Ba_3-xLa_xCo₂Fe₂₄O₄₁[J]. 吉林大学学报(工学版), 2006, 36(06): 856 -0860 .
[4]	Zhang Quan-fa，Li Ming-zhe，Sun Gang，Ge Xin . Comparison between flexible and rigid blank-holding in multi-point forming[J]. 吉林大学学报(工学版), 2007, 37(01): 25 -30 .
[5]	Yang Shu-kai, Song Chuan-xue, An Xiao-juan, Cai Zhang-lin . Analyzing effects of suspension bushing elasticity on vehicle yaw response character with virtual prototype method[J]. 吉林大学学报(工学版), 2007, 37(05): 994 -0999 .
[6]	. [J]. 吉林大学学报(工学版), 2007, 37(06): 1284 -1287 .
[7]	Che Xiang-jiu，Liu Da-you，Wang Zheng-xuan . Construction of joining surface with G¹ continuity for two NURBS surfaces[J]. 吉林大学学报(工学版), 2007, 37(04): 838 -841 .
[8]	Liu Han-bing, Jiao Yu-ling, Liang Chun-yu,Qin Wei-jun . Effect of shape function on computing precision in meshless methods[J]. 吉林大学学报(工学版), 2007, 37(03): 715 -0720 .
[9]	Li Yue-ying，Liu Yong-bing，Chen Hua . Surface hardening and tribological properties of a cam materials[J]. 吉林大学学报(工学版), 2007, 37(05): 1064 -1068 .
[10]	Feng Hao，Xi Jian-feng，Jiao Cheng-wu . Placement of roadside traffic signs based on visibility distance[J]. 吉林大学学报(工学版), 2007, 37(04): 782 -785 .

Merging guidance of exclusive lanes for connected and autonomous vehicles based on deep reinforcement learning

RICH HTML

PDF (PC)