基于多模态BEV特征融合的三维目标检测算法

doi:10.13229/j.cnki.jdxbgxb.20250498

Abstract

Abstract:

Multi-sensor fusion is crucial for achieving accurate and reliable autonomous driving. However， current point-level fusion methods enhance LiDAR point cloud features by projecting camera features onto LiDAR data， a process that often discards semantic information from camera features due to projection errors. To address the loss of local semantic information in images and the introduction of excessive background points during keypoint sampling， we propose an enhanced algorithm based on the BEVFusion network （a simple and robust LiDAR-Camera fusion framework）. Specifically， dilated convolutions and a weighted bidirectional feature pyramid are introduced in the image stream to expand the perceptual field while preserving semantic integrity. Additionally， a semantic-aware farthest point sampling （SA-FPS） mechanism is proposed to prioritize foreground points with richer information. Building on the dynamic fusion framework， we integrate squeeze-and-excitation （SE） operations and design an adaptive gated dual-dynamic fusion network to dynamically select valuable information from different modalities. Experimental evaluations on the KITTI dataset and generalization tests on the nuScenes dataset demonstrate that our algorithm achieves superior detection accuracy compared to state-of-the-art methods.

Key words: 3D object detection, farthest point sampling （FPS）, weighted bidirectional feature pyramid, multi-modal fusion, dual-dynamic fusion

CLC Number:

TP391

Hua CAI,Wei-wei SUN,Hai-feng ZHANG,Gao-peng ZHANG,Qiang FU,Ting-ting KOU,Yu-tong HAN. Multi-modal Bird's Eye View （BEV） feature fusion for 3D object detection[J].Journal of Jilin University(Engineering and Technology Edition), 2026, 56(10): 2648-2661.

Figures/Tables 13

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Table 2

Table 3

Fig.6

Fig.7

Fig.8

Fig.9

Table 4

References 44

[1]	Chen Y L, Liu S, Shen X Y, et al. DSGN: Deep stereo geometry network for 3D object detection[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 12533-12542.
[2]	Li P L, Chen X Z, Shen S J. Stereo R-CNN based 3D object detection for autonomous driving[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 7636-7644.
[3]	Sun J M, Chen L H, Xie Y M, et al. Disp R-CNN: Stereo 3D object detection via shape prior guided instance disparity estimation[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 10545-10554.
[4]	才华, 郑延阳, 付强, 等. 基于多尺度候选融合与优化的三维目标检测算法[J]. 吉林大学学报: 工学版,2025, 55(2): 709-721.
	Cai Hua, Zheng Yan-yang, Fu Qiang, et al. A 3D object detection algorithm based on multi-scale candidate fusion and optimization[J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(2): 709-721.
[5]	Meyer P G, Laddha A, Kee E, et al. LaserNet: an efficient probabilistic 3D object detector for autonomous driving[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 12669-12678.
[6]	Yin Z, Sun P, Zhang Y, et al. End-to-End multi-view fusion for 3D object detection in LiDAR point clouds[J]. Conference on Robot Learning,2019,100:923-932.
[7]	Wang Z X, Jia K. Frustum ConvNet: sliding frustums to aggregate local point-wise features for amodal 3D object detection[C]∥2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, Macau, China, 2019:1742-1749.
[8]	Shin K, Kwon Y P, Tomizuka M. RoarNet: a robust 3D object detection based on region approximation refinement[C]∥IEEE Intelligent Vehicles Symposium, Paris, France 2019: 2510-2515.
[9]	Du X X, Ang M H, Karaman S, et al. A general pipeline for 3D detection of vehicles[C]∥IEEE International Conference on Robotics and Automation, Brisbane, Australia, 2018: 3194-3200.
[10]	Yoo J H, Kim Y, Kim J, et al. 3D-CVF: generating joint camera and LiDAR features using cross-view spatial feature fusion for 3D object detection[C]∥16th European Conference, Glasgow, UK, 2020: 720-736.
[11]	Zhang Y C, Guo Y, Niu H B, et al. SimpleFusion: 3D object detection by fusing RGB images and point clouds[C]∥International Conference on Pattern Recognition, Machine Vision and Intelligent Algorithms, Beihai, China, 2023: 47-51.
[12]	Bai X Y, Hu Z Y, Zhu X G, et al. TransFusion: Robust LiDAR-camera fusion for 3D object detection with transformers[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA,2022: 1080-1089.
[13]	Pang S, Morris D, Radha H. CLOCs: Camera-LiDAR object candidates fusion for 3D object detection[C]∥IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, USA, 2020: 10386-10393.
[14]	Qi C R, Liu W, Wu C X, et al. Frustum pointNets for 3D object detection from RGB-D Data[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 918-927.
[15]	Liang T T, Xie H W, Yu K C, et al. BEVFusion: a simple and robust LiDAR-camera fusion framework[C]∥Proc IEEE Conf Comput Vis Pattern Recognit, 2022: 10421-10434.
[16]	Deng J J, Shi S S, Li P W, et al. Voxel R-CNN: towards high performance voxel-based 3D object detection[C]∥The Thirty-Fifth AAAI Conference on Artificial Intelligence, British Columbia, Canada,2021: 1201-1209.
[17]	Zhou Y, Tuzel O. VoxelNet: end-to-end learning for point cloud based 3D object detection[C]∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 4490-4499.
[18]	Yan Y, Mao Y X, Li B. SECOND: sparsely embedded convolutional detection[J]. Sensors, 2018, 18(10): 3337.
[19]	Shi S S, Guo C X, Jiang L, et al. PV-RCNN: point-voxel feature set abstraction for 3D object detection[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA,2020: 10526-10535.
[20]	Chen X Z, Ma H M, Wan J, et al. Multi-view 3D object detection network for autonomous driving[C]∥2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017:6526-6534.
[21]	Ku J, Mozifian M, Lee J, et al. Joint 3D proposal generation and object detection from view aggregation[C]∥2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, Madrid, Spain, 2018: 1-8.
[22]	Huang T T, Liu Z, Chen X W, et al. EPNet: enhancing point features with image semantics for 3d object detection[C]∥Computer Vision-ECCV 2020, Glasgow, UK, 2020: 35-52.
[23]	Jiao Y, Jie Z Q, Chen S X, et al. MSMDFusion: fusing LiDAR and camera at multiple scales with multi-depth seeds for 3D object detection[C]∥2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, Canada,2023: 21643-21652.
[24]	Liu Z W, Cheng J R, Fan J, et al. Multi-modal fusion based on depth adaptive mechanism for 3D object detection[J].IEEE Transactions on Multimedia, 2025,27: 707-717.
[25]	Zhang Y N, Chen J X, Huang D. CAT-Det: contrastively augmented transformer for multimodal 3D object detection[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 898-907.
[26]	Liao B C, Chen S Y, Wang X G, et al. MapTR: structured modeling and learning for online vectorized HD map construction[C]∥International Conference on Learning Representations, Kigali, Rwanda, 2023: 1-18.
[27]	Liu Z J, Tang H T, Amini A, et al. BEVFusion: multi-task multi-sensor fusion with unified bird's-eye view representation[C]∥IEEE International Conference on Robotics and Automation, London, United Kingdom,2023: 2774-2781.
[28]	Li Z Q, Wang W H, Li H Y, et al. BEVFormer: learning Bird's-Eye-View representation from multi-camera images via spatiotemporal transformers[C]∥Computer Vision-ECCV 2022, Tel Aviv, Israel,2022:1-18.
[29]	Yang L, Yu K C, Tang T,et al. BEVHeight: a robust framework for vision-based roadside 3D object detection[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, Canada, 2023: 21611-21620.
[30]	Geiger A, Lenz P, Urtasun R. Are we ready for autonomous driving? The KITTI Vision Benchmark Suite[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Providence, USA, 2012: 3354-3361.
[31]	Caesar H, Bankiti V, Lang A H,et al. nuScenes: a multimodal dataset for autonomous driving[C]∥2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), Seattle, USA,2020: 11618-11628.
[32]	Lang A H, Vora S, Caesar H, et al. PointPillars: fast encoders for object detection from point clouds[C]∥2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA 2019: 12689-12697.
[33]	Shi S S, Wang X G, Li H S. PointRCNN: 3D object proposal generation and detection from point cloud[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA 2019: 770-779.
[34]	Xu S Q, Zhou D F, Fang J, et al. FusionPainting: multimodal fusion with adaptive attention for 3D object detection[C]∥IEEE International Intelligent Transportation Systems Conference, Indianapolis, USA,2021: 3047-3054.
[35]	Liu Z, Huang T T, Li B L, et al. EPNet++: Cascade Bi-directional fusion for multi-modal 3D object detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023,45(7): 8324-8341.
[36]	Song Z Y, Zhang G X, Xie J, et al. VoxelNextFusion: a simple, unified, and effective voxel fusion framework for multimodal 3-D object detection[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-12.
[37]	Huang J J, Huang G, Zhu Z, et al. BEVDet: high-performance multi-camera 3D object detection in bird-eye-view[J]. arXiv, 2021: abs/2112.11790.
[38]	Yin T W, Zhou X Y, Krahenbuhl P. Center-Based 3D object detection and tracking[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 11784-11793.
[39]	Yoo J H, Kim Y, Kim J, et al. 3D-CVF: Generating joint camera and LiDAR features using cross-view spatial feature fusion for 3D object detection[C]∥Computer Vision-ECCV 2020, Glasgow,UK, 2020: 720-736.
[40]	Bai X Y, Hu Z Y, Zhu X G, et al. TransFusion: robust LiDAR-camera fusion for 3D object detection with transformers[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 1080-1089.
[41]	Xie E Z, Yu Z D, Zhou D Q, et al. M2 BEV: multi-camera joint 3D detection and segmentation with unified birds-eye view representation[J/OL]. [2024-10-22]. .
[42]	Yin T W, Zhou X Y, Krahenbuhl P. Multimodal virtual point 3D detection[C]∥Neural Information Processing Systems, 2021: 16494-16507.
[43]	Chen Z H, Li Z Y, Zhang S Q, et al. AutoAlign: Pixel-Instance feature aggregation for multimodal 3D object detection[C]∥International Joint Conference on Artificial Intelligence, Vienna, Austria, 2022:827-833.
[44]	Chen X Y, Zhang T Y, Wang Y, et al. FUTR3D: a unified sensor fusion framework for 3D detection[C]∥IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Vancouver, Canada, 2023: 172-181.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

算法	数据	3D Car（IoU=0.7）			3D Ped.（IoU=0.5）			3D Cyc.（IoU=0.5）
算法	数据	简单	中等	困难	简单	中等	困难	简单	中等	困难
SECOND^［18］	L	84.65	75.96	68.71	45.31	35.52	33.14	75.83	60.82	53.67
PointPillars^［32］	L	82.58	74.31	68.99	51.45	41.92	38.89	77.10	58.65	51.92
PointRCNN^［33］	L	86.96	75.64	70.70	47.98	39.37	36.01	74.96	58.82	52.53
EPNet^［22］	L&C	89.81	79.28	74.59	-	-	-	-	-	-
PointPainting^［34］	L&C	82.11	71.70	67.08	50.32	40.97	37.84	77.63	63.78	55.89
EPNet++^［35］	L&C	91.37	81.96	76.71	52.79	44.38	41.29	76.15	59.17	53.67
VoxelNextFusion^［36］	L&C	90.90	82.93	80.62	53.27	47.86	42.11	78.56	65.27	54.24
Ours	L&C	90.98	82.99	80.69	53.51	48.10	42.21	78.76	65.52	54.56

算法	数据	map	NDS	Car	Truck	C.V.	Bus	Trailer	Barrier	Motor.	Bicycle	Ped.	T.C.
BEVDet^［37］	C	42.4	47.6	64.3	35.0	16.2	35.8	35.4	61.4	44.8	29.6	41.4	60.1
BEVFormer^［28］	C	48.1	56.9	67.7	39.2	22.9	35.7	39.6	62.5	47.9	40.7	54.4	70.3
BEVHeight^［29］	C	53.2	61.0	68.6	44.8	27.4	42.8	48.5	69.8	54.2	45.9	57.7	72.6
CenterPoint^［38］	L	60.3	67.3	85.2	53.5	20.0	63.6	56.6	71.1	59.5	30.7	84.6	78.4
3D-CVF^［39］	L+C	52.7	62.4	83.3	45.1	15.7	48.6	49.5	65.7	51.2	30.6	74.1	62.9
TransFusion^［40］	L+C	68.9	71.5	87.5	59.9	33.0	68.1	60.9	78.1	73.5	52.9	88.4	86.7
BEVFusion^［27］	L+C	70.2	72.3	88.6	60.1	39.3	69.8	63.8	80.0	74.1	51.0	89.2	86.5
BEVFusion^［15］	L+C	71.3	73.4	88.3	70.0	34.3	69.1	62.1	78.5	72.1	52.0	89.2	86.7
Ours	L+C	71.9	73.9	89.0	70.5	34.8	69.5	62.7	78.7	72.3	52.3	89.5	86.9

算法	数据	map（test）	NDS（test）	map（val）	NDS（val）	MACS（G）	Latency
M²BEV［41］	C	42.9	47.4	41.7	47.0	-	-
BEVFormer^［28］	C	44.5	53.5	41.6	51.7	-	-
PointPillars^［32］	L	-	-	52.3	61.3	65.5	34.4
SECOND^［18］	L	52.8	63.3	52.6	63.0	85.0	69.8
CenterPoint^［38］	L	60.3	67.3	59.6	66.8	153.5	80.7
MVP^［42］	C+L	66.4	70.5	66.1	70.0	371.7	187.1
AutoAlign^［43］	C+L	-	-	66.6	71.1	-	-
FUTR3D^［44］	C+L	-	-	64.5	68.3	1069.0	321.4
TransFusion^［40］	C+L	68.9	71.6	67.5	71.3	485.8	156.6
BEVFusion^［15］	C+L	70.2	72.9	68.5	71.4	253.2	119.2
Ours	C+L	71.1	73.5	69.2	71.8	223.1	110.6

算法	双向跨尺度连接和加权特征融合	双动态融合	基于语义感知最远点采样	map （LEVEL1）	map （LEVEL2）
基础网络（BEVFusion）				85.70	80.80
实验网络1	√			85.87	80.96
实验网路2	√	√		85.95	81.10
本文	√	√	√	85.21	81.25

Multi-modal Bird's Eye View （BEV） feature fusion for 3D object detection

RICH HTML

PDF (PC)

Abstract

Cite this article

share this article

Figures/Tables 13

References 44

Related Articles 4

Metrics

Comments

Recommended 0

[1]	Hua CAI,Yan-yang ZHENG,Qiang FU,Sheng-yu WANG,Wei-gang WANG,Zhi-yong MA. Three-dimensional object detection algorithm based on multi-scale candidate fusion and optimization [J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(2): 709-721.
[2]	Xin CHENG,Sheng-xian LIU,Jing-mei ZHOU,Zhou ZHOU,Xiang-mo ZHAO. 3D object detection algorithm fusing dense connectivity and Gaussian distance [J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(12): 3589-3600.
[3]	Wei LYU,Jia-ze HAN,Jing-hui CHU,Pei-guang JING. Multi⁃modal self⁃attention network for video memorability prediction [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(4): 1211-1219.
[4]	Bo TAO,Fu-wu YAN,Zhi-shuai YIN,Dong-mei WU. 3D object detection based on high⁃precision map enhancement [J]. Journal of Jilin University(Engineering and Technology Edition), 2023, 53(3): 802-809.