基于多模态BEV特征融合的三维目标检测算法

doi:10.13229/j.cnki.jdxbgxb.20250498

摘要/Abstract

摘要：

为了改善多传感器融合过程中图像局部语义信息丢失和关机键点采样过程中引入大量背景点的问题，提出了一种基于BEVFusion网络的改进算法，在图像流中引入空洞卷积和加权双向特征金字塔，扩张感知区域，维持语义完整性；引入语义感知的最远点采样算法，关注信息量更大的前景点；在动态融合网络基础上引入了挤压和激励操作，设计了自适应门控的双动态融合网络，动态地选择不同模态有价值的信息。本文使用KITTI 数据集进行实验评估，并在 nuScenes 数据集上进行了泛化实验。实验结果证明，本文算法与当前主流算法相比表现出更高的检测精度。

关键词: 三维目标检测, 最远点采样, 加权双向特征金字塔, 多模态融合, 双动态融合

Abstract:

Multi-sensor fusion is crucial for achieving accurate and reliable autonomous driving. However， current point-level fusion methods enhance LiDAR point cloud features by projecting camera features onto LiDAR data， a process that often discards semantic information from camera features due to projection errors. To address the loss of local semantic information in images and the introduction of excessive background points during keypoint sampling， we propose an enhanced algorithm based on the BEVFusion network （a simple and robust LiDAR-Camera fusion framework）. Specifically， dilated convolutions and a weighted bidirectional feature pyramid are introduced in the image stream to expand the perceptual field while preserving semantic integrity. Additionally， a semantic-aware farthest point sampling （SA-FPS） mechanism is proposed to prioritize foreground points with richer information. Building on the dynamic fusion framework， we integrate squeeze-and-excitation （SE） operations and design an adaptive gated dual-dynamic fusion network to dynamically select valuable information from different modalities. Experimental evaluations on the KITTI dataset and generalization tests on the nuScenes dataset demonstrate that our algorithm achieves superior detection accuracy compared to state-of-the-art methods.

Key words: 3D object detection, farthest point sampling （FPS）, weighted bidirectional feature pyramid, multi-modal fusion, dual-dynamic fusion

中图分类号:

TP391

才华,孙微微,张海峰,张高鹏,付强,寇婷婷,韩雨桐. 基于多模态BEV特征融合的三维目标检测算法[J]. 吉林大学学报(工学版), 2026, 56(10): 2648-2661.

Hua CAI,Wei-wei SUN,Hai-feng ZHANG,Gao-peng ZHANG,Qiang FU,Ting-ting KOU,Yu-tong HAN. Multi-modal Bird's Eye View （BEV） feature fusion for 3D object detection[J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(10): 2648-2661.

图/表 13

图1

图2

图3

图4

图5

表1

表2

图6

图7

图8

图9

表4

参考文献 44

[1]	Chen Y L, Liu S, Shen X Y, et al. DSGN: Deep stereo geometry network for 3D object detection[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 12533-12542.
[2]	Li P L, Chen X Z, Shen S J. Stereo R-CNN based 3D object detection for autonomous driving[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 7636-7644.
[3]	Sun J M, Chen L H, Xie Y M, et al. Disp R-CNN: Stereo 3D object detection via shape prior guided instance disparity estimation[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 10545-10554.
[4]	才华, 郑延阳, 付强, 等. 基于多尺度候选融合与优化的三维目标检测算法[J]. 吉林大学学报: 工学版,2025, 55(2): 709-721.
	Cai Hua, Zheng Yan-yang, Fu Qiang, et al. A 3D object detection algorithm based on multi-scale candidate fusion and optimization[J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(2): 709-721.
[5]	Meyer P G, Laddha A, Kee E, et al. LaserNet: an efficient probabilistic 3D object detector for autonomous driving[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 12669-12678.
[6]	Yin Z, Sun P, Zhang Y, et al. End-to-End multi-view fusion for 3D object detection in LiDAR point clouds[J]. Conference on Robot Learning,2019,100:923-932.
[7]	Wang Z X, Jia K. Frustum ConvNet: sliding frustums to aggregate local point-wise features for amodal 3D object detection[C]∥2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, Macau, China, 2019:1742-1749.
[8]	Shin K, Kwon Y P, Tomizuka M. RoarNet: a robust 3D object detection based on region approximation refinement[C]∥IEEE Intelligent Vehicles Symposium, Paris, France 2019: 2510-2515.
[9]	Du X X, Ang M H, Karaman S, et al. A general pipeline for 3D detection of vehicles[C]∥IEEE International Conference on Robotics and Automation, Brisbane, Australia, 2018: 3194-3200.
[10]	Yoo J H, Kim Y, Kim J, et al. 3D-CVF: generating joint camera and LiDAR features using cross-view spatial feature fusion for 3D object detection[C]∥16th European Conference, Glasgow, UK, 2020: 720-736.
[11]	Zhang Y C, Guo Y, Niu H B, et al. SimpleFusion: 3D object detection by fusing RGB images and point clouds[C]∥International Conference on Pattern Recognition, Machine Vision and Intelligent Algorithms, Beihai, China, 2023: 47-51.
[12]	Bai X Y, Hu Z Y, Zhu X G, et al. TransFusion: Robust LiDAR-camera fusion for 3D object detection with transformers[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA,2022: 1080-1089.
[13]	Pang S, Morris D, Radha H. CLOCs: Camera-LiDAR object candidates fusion for 3D object detection[C]∥IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, USA, 2020: 10386-10393.
[14]	Qi C R, Liu W, Wu C X, et al. Frustum pointNets for 3D object detection from RGB-D Data[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 918-927.
[15]	Liang T T, Xie H W, Yu K C, et al. BEVFusion: a simple and robust LiDAR-camera fusion framework[C]∥Proc IEEE Conf Comput Vis Pattern Recognit, 2022: 10421-10434.
[16]	Deng J J, Shi S S, Li P W, et al. Voxel R-CNN: towards high performance voxel-based 3D object detection[C]∥The Thirty-Fifth AAAI Conference on Artificial Intelligence, British Columbia, Canada,2021: 1201-1209.
[17]	Zhou Y, Tuzel O. VoxelNet: end-to-end learning for point cloud based 3D object detection[C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 4490-4499.
[18]	Yan Y, Mao Y X, Li B. SECOND: sparsely embedded convolutional detection[J]. Sensors, 2018, 18(10): 3337.
[19]	Shi S S, Guo C X, Jiang L, et al. PV-RCNN: point-voxel feature set abstraction for 3D object detection[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA,2020: 10526-10535.
[20]	Chen X Z, Ma H M, Wan J, et al. Multi-view 3D object detection network for autonomous driving[C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017:6526-6534.
[21]	Ku J, Mozifian M, Lee J, et al. Joint 3D proposal generation and object detection from view aggregation[C]∥2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, Madrid, Spain, 2018: 1-8.
[22]	Huang T T, Liu Z, Chen X W, et al. EPNet: enhancing point features with image semantics for 3d object detection[C]∥Computer Vision-ECCV 2020, Glasgow, UK, 2020: 35-52.
[23]	Jiao Y, Jie Z Q, Chen S X, et al. MSMDFusion: fusing LiDAR and camera at multiple scales with multi-depth seeds for 3D object detection[C]∥2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, Canada,2023: 21643-21652.
[24]	Liu Z W, Cheng J R, Fan J, et al. Multi-modal fusion based on depth adaptive mechanism for 3D object detection[J].IEEE Transactions on Multimedia, 2025,27: 707-717.
[25]	Zhang Y N, Chen J X, Huang D. CAT-Det: contrastively augmented transformer for multimodal 3D object detection[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 898-907.
[26]	Liao B C, Chen S Y, Wang X G, et al. MapTR: structured modeling and learning for online vectorized HD map construction[C]∥International Conference on Learning Representations, Kigali, Rwanda, 2023: 1-18.
[27]	Liu Z J, Tang H T, Amini A, et al. BEVFusion: multi-task multi-sensor fusion with unified bird's-eye view representation[C]∥IEEE International Conference on Robotics and Automation, London, United Kingdom,2023: 2774-2781.
[28]	Li Z Q, Wang W H, Li H Y, et al. BEVFormer: learning Bird's-Eye-View representation from multi-camera images via spatiotemporal transformers[C]∥Computer Vision-ECCV 2022, Tel Aviv, Israel,2022:1-18.
[29]	Yang L, Yu K C, Tang T,et al. BEVHeight: a robust framework for vision-based roadside 3D object detection[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, Canada, 2023: 21611-21620.
[30]	Geiger A, Lenz P, Urtasun R. Are we ready for autonomous driving? The KITTI Vision Benchmark Suite[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Providence, USA, 2012: 3354-3361.
[31]	Caesar H, Bankiti V, Lang A H,et al. nuScenes: a multimodal dataset for autonomous driving[C]∥2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Seattle, USA,2020: 11618-11628.
[32]	Lang A H, Vora S, Caesar H, et al. PointPillars: fast encoders for object detection from point clouds[C]∥2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA 2019: 12689-12697.
[33]	Shi S S, Wang X G, Li H S. PointRCNN: 3D object proposal generation and detection from point cloud[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA 2019: 770-779.
[34]	Xu S Q, Zhou D F, Fang J, et al. FusionPainting: multimodal fusion with adaptive attention for 3D object detection[C]∥IEEE International Intelligent Transportation Systems Conference, Indianapolis, USA,2021: 3047-3054.
[35]	Liu Z, Huang T T, Li B L, et al. EPNet++: Cascade Bi-directional fusion for multi-modal 3D object detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023,45(7): 8324-8341.
[36]	Song Z Y, Zhang G X, Xie J, et al. VoxelNextFusion: a simple, unified, and effective voxel fusion framework for multimodal 3-D object detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-12.
[37]	Huang J J, Huang G, Zhu Z, et al. BEVDet: high-performance multi-camera 3D object detection in bird-eye-view[J]. arXiv, 2021: abs/2112.11790.
[38]	Yin T W, Zhou X Y, Krahenbuhl P. Center-Based 3D object detection and tracking[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 11784-11793.
[39]	Yoo J H, Kim Y, Kim J, et al. 3D-CVF: Generating joint camera and LiDAR features using cross-view spatial feature fusion for 3D object detection[C]∥Computer Vision-ECCV 2020, Glasgow,UK, 2020: 720-736.
[40]	Bai X Y, Hu Z Y, Zhu X G, et al. TransFusion: robust LiDAR-camera fusion for 3D object detection with transformers[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 1080-1089.
[41]	Xie E Z, Yu Z D, Zhou D Q, et al. M2 BEV: multi-camera joint 3D detection and segmentation with unified birds-eye view representation[J/OL]. [2024-10-22]. .
[42]	Yin T W, Zhou X Y, Krahenbuhl P. Multimodal virtual point 3D detection[C]∥Neural Information Processing Systems, 2021: 16494-16507.
[43]	Chen Z H, Li Z Y, Zhang S Q, et al. AutoAlign: Pixel-Instance feature aggregation for multimodal 3D object detection[C]∥International Joint Conference on Artificial Intelligence, Vienna, Austria, 2022:827-833.
[44]	Chen X Y, Zhang T Y, Wang Y, et al. FUTR3D: a unified sensor fusion framework for 3D detection[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Vancouver, Canada, 2023: 172-181.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

算法	数据	3D Car（IoU=0.7）			3D Ped.（IoU=0.5）			3D Cyc.（IoU=0.5）
算法	数据	简单	中等	困难	简单	中等	困难	简单	中等	困难
SECOND^［18］	L	84.65	75.96	68.71	45.31	35.52	33.14	75.83	60.82	53.67
PointPillars^［32］	L	82.58	74.31	68.99	51.45	41.92	38.89	77.10	58.65	51.92
PointRCNN^［33］	L	86.96	75.64	70.70	47.98	39.37	36.01	74.96	58.82	52.53
EPNet^［22］	L&C	89.81	79.28	74.59	-	-	-	-	-	-
PointPainting^［34］	L&C	82.11	71.70	67.08	50.32	40.97	37.84	77.63	63.78	55.89
EPNet++^［35］	L&C	91.37	81.96	76.71	52.79	44.38	41.29	76.15	59.17	53.67
VoxelNextFusion^［36］	L&C	90.90	82.93	80.62	53.27	47.86	42.11	78.56	65.27	54.24
Ours	L&C	90.98	82.99	80.69	53.51	48.10	42.21	78.76	65.52	54.56

算法	数据	map	NDS	Car	Truck	C.V.	Bus	Trailer	Barrier	Motor.	Bicycle	Ped.	T.C.
BEVDet^［37］	C	42.4	47.6	64.3	35.0	16.2	35.8	35.4	61.4	44.8	29.6	41.4	60.1
BEVFormer^［28］	C	48.1	56.9	67.7	39.2	22.9	35.7	39.6	62.5	47.9	40.7	54.4	70.3
BEVHeight^［29］	C	53.2	61.0	68.6	44.8	27.4	42.8	48.5	69.8	54.2	45.9	57.7	72.6
CenterPoint^［38］	L	60.3	67.3	85.2	53.5	20.0	63.6	56.6	71.1	59.5	30.7	84.6	78.4
3D-CVF^［39］	L+C	52.7	62.4	83.3	45.1	15.7	48.6	49.5	65.7	51.2	30.6	74.1	62.9
TransFusion^［40］	L+C	68.9	71.5	87.5	59.9	33.0	68.1	60.9	78.1	73.5	52.9	88.4	86.7
BEVFusion^［27］	L+C	70.2	72.3	88.6	60.1	39.3	69.8	63.8	80.0	74.1	51.0	89.2	86.5
BEVFusion^［15］	L+C	71.3	73.4	88.3	70.0	34.3	69.1	62.1	78.5	72.1	52.0	89.2	86.7
Ours	L+C	71.9	73.9	89.0	70.5	34.8	69.5	62.7	78.7	72.3	52.3	89.5	86.9

算法	数据	map（test）	NDS（test）	map（val）	NDS（val）	MACS（G）	Latency
M²BEV［41］	C	42.9	47.4	41.7	47.0	-	-
BEVFormer^［28］	C	44.5	53.5	41.6	51.7	-	-
PointPillars^［32］	L	-	-	52.3	61.3	65.5	34.4
SECOND^［18］	L	52.8	63.3	52.6	63.0	85.0	69.8
CenterPoint^［38］	L	60.3	67.3	59.6	66.8	153.5	80.7
MVP^［42］	C+L	66.4	70.5	66.1	70.0	371.7	187.1
AutoAlign^［43］	C+L	-	-	66.6	71.1	-	-
FUTR3D^［44］	C+L	-	-	64.5	68.3	1069.0	321.4
TransFusion^［40］	C+L	68.9	71.6	67.5	71.3	485.8	156.6
BEVFusion^［15］	C+L	70.2	72.9	68.5	71.4	253.2	119.2
Ours	C+L	71.1	73.5	69.2	71.8	223.1	110.6

算法	双向跨尺度连接和加权特征融合	双动态融合	基于语义感知最远点采样	map （LEVEL1）	map （LEVEL2）
基础网络（BEVFusion）				85.70	80.80
实验网络1	√			85.87	80.96
实验网路2	√	√		85.95	81.10
本文	√	√	√	85.21	81.25