基于特征融合注意力的RGB-D语义分割算法

doi:10.13229/j.cnki.jdxbgxb.20250159

摘要/Abstract

摘要：

针对室内场景感知任务中，现有的多模态融合方法主要侧重于单一模态（如RGB或深度图像）的注意力，而对模态融合后联合特征空间中的注意力缺乏有效利用，且许多网络的参数量较大，在参数量减少时模型精度通常会显著下降等问题，提出了一种多模态语义分割模型，该模型通过ConvNeXt和MiT分别提取RGB特征和深度特征，并引入跨模态特征校正模块（CMFCM），通过空间和通道特征增强促进模态间的交互与校正。此外，模型还设计了跨模态注意力融合模块（CMAFM），引入了联合特征空间中的注意力，实现多尺度特征的有效融合，充分利用两种模态的分割信息，同时通过深度可分离卷积显著降低计算成本。最终，在NYU-Depth V2数据集上的实验表明，与CMX、Dformer和SA-Gate模型相比，该方法在分割精度提升1%的同时参数量减少了15.1%，显著降低了计算复杂度。

关键词: 语义分割, 多模态融合, 室内感知, 注意力机制

Abstract:

A multimodal semantic segmentation model was proposed for indoor scene perception tasks. Existing multimodal fusion methods mainly focus on a single modality， such as RGB or depth image attention， while lacking effective utilization of attention in the joint feature space after modality fusion. Additionally， many networks suffer from a significant drop in accuracy when the number of parameters is reduced， due to their large parameter sizes. The RGB and depth features are extracted using ConvNeXt and MiT， respectively. A Cross-modal Feature Correction Module （CMFCM） is introduced to enhance spatial and channel features， facilitating interaction and correction between modalities. Additionally， a Cross-modal Attention Fusion Module （CMAFM） is designed to apply attention in the joint feature space， enabling effective multi-scale feature fusion and maximizing the utilization of segmentation information from both modalities. Depthwise separable convolution is also employed to significantly reduce computational costs. Finally， experiments on the NYU-Depth V2 dataset show that， compared with CMX， Dformer， and SA-Gate models， the proposed method improves segmentation accuracy by 1% while reducing the number of parameters by 15.1%， significantly lowering computational complexity.

Key words: semantic segmentation, multimodal fusion, indoor perception, attention mechanism

中图分类号:

TP301.6

江晟,路启,夏淼磊,王稼禹. 基于特征融合注意力的RGB-D语义分割算法[J]. 吉林大学学报(工学版), 2026, 56(9): 2435-2443.

Sheng JIANG,Qi LU,Miao-lei XIA,Jia-yu WANG. RGB⁃D semantic segmentation algorithm based on feature fusion attention[J]. Journal of Jilin University(Engineering and Technology Edition), 2026, 56(9): 2435-2443.

图/表 9

图1

图2

图3

图4

图5

图6

表1

图7

表2

参考文献 18

[1]	王文俊. 基于深度卷积神经网络的点云数据语义分割方法研究[D].成都:电子科技大学信息与通信工程学院, 2024.
	Wang Wen-jun. Research on semantic segmentation of point cloud data based on deep convolutional neural networks[D].Chengdu: School of Information and Communication Engineering, University of Electronic Science and Technology of China, 2024.
[2]	Jiang Jin-dong, Zheng Lu-nan, Luo Fei, et al. Rednet: residual encoder-decoder network for indoor RGB-D semantic segmentation[J/OL]. [2025-02-10]. .
[3]	Seichter D, Köhler M, Lewandowski B, et al. Efficient RGB-D semantic segmentation for indoor scene analysis[C]∥2021 IEEE International Conference on Robotics and Automation, Xi'an, China, 2021: 13525-13531.
[4]	Du S Q, Tang S J, Wang W X, et al. PSCNET: efficient RGB-D semantic segmentation parallel network based on spatial and channel attention[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2022, 1: 129-136.
[5]	Chen X, Lin K Y, Wang J, et al. Bi-directional cross-modality feature propagation with separation-and-aggregation gate for RGB-D semantic segmentation[C]∥European Conference on Computer Vision, Glasgow, UK, 2020: 561-577.
[6]	Zhang Jia-ming, Liu Hua-yao, Yang Kai-lun, et al. CMX: cross-modal fusion for RGB-X semantic segmentation with transformers[J]. IEEE Transactions on Intelligent Transportation Systems, 2023,24(12): 14679-14694.
[7]	Yin Bo-wen, Zhang Xu-ying, Li Zhong-yu, et al. Dformer: rethinking RGBD representation learning for semantic segmentation[J/OL].[2025-02-10]. .
[8]	Bui M, Alexis K. Diffusion-based RGB-D semantic segmentation with deformable attention transformer[J/OL].[2025-02-10]. .
[9]	Dong S, Feng Y, Yang Q, et al. Efficient multimodal semantic segmentation via dual-prompt learning[C]∥2024 IEEE/RSJ International Conference on Intelligent Robots and Systems, Abu Dhabi, UAE, 2024: 14196-14203.
[10]	缪君, 严杰, 杜荣华, 等.基于双向特征融合的物体位姿估计方法[J].吉林大学学报: 工学版, 2026, 56(2): 523-532.
	Miao Jun, Yan Jie, Du Rong-hua, et al. Object pose estimation method based on bidirectional feature fusion[J]. Journal of Jilin University (Engineering and Technology Edition), 2026, 56(2): 523-532.
[11]	王雪, 李占山, 吕颖达.基于多尺度感知和语义适配的医学图像分割算法[J].吉林大学学报: 工学版,2022, 52(3): 640-647.
	Wang Xue, Li Zhan-shan, Ying-da Lyu. Medical image segmentation algorithm based on multi-scale perception and semantic adaptation[J]. Journal of Jilin University (Engineering and Technology Edition),2022, 52(3): 640-647.
[12]	周大可, 张超, 杨欣. 基于多尺度特征融合及双重注意力机制的自监督三维人脸重建[J].吉林大学学报: 工学版, 2022, 52(10): 2428-2437.
	Zhou Da-ke, Zhang Chao, Yang Xin. Self-supervised 3D face reconstruction based on multi-scale feature fusion and dual attention mechanism[J]. Journal of Jilin University (Engineering and Technology Edition), 2022, 52(10): 2428-2437.
[13]	Xie E, Wang W, Yu Z, et al. SegFormer: simple and efficient design for semantic segmentation with transformers[J]. Advances in Neural Information Processing Systems, 2021, 34: 12077-12090.
[14]	Liu Z, Mao H, Wu C Y, et al. A convnet for the 2020s[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 11976-11986.
[15]	Song Y, Wen J, Liu D, et al. Deep robotic grasping prediction with hierarchical RGB-D fusion[J]. International Journal of Control, Automation and Systems, 2022, 20(1): 243-254.
[16]	Liu Z, Tan Y, He Q, et al. SwinNet: swin transformer drives edge-aware RGB-D and RGB-T salient object detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(7): 4486-4497.
[17]	钱白云, 吕朝阳, 张维宁, 等. 基于多传感器信息融合与混合感受野残差卷积神经网络的调相机转子故障诊断[J].计算机测量与控制, 2023, 31(9): 29-35.
	Qian Bai-yun, Lv Chao-yang, Zhang Wei-ning, et al. Phase condenser rotor fault diagnosis based on multi-sensor information fusion and mixed receptive field residual convolutional neural network[J]. Computer Measurement & Control, 2023, 31(9): 29-35.
[18]	孙启超, 恩擎, 段立娟, 等. 基于多模态自适应卷积的RGB-D图像语义分割[J]. 计算机辅助设计与图形学学报, 2022, 34(8): 1272-1282.
	Sun Qi-chao, Qing En, Duan Li-juan, et al. RGB-D image semantic segmentation based on multimodal adaptive convolution[J]. Journal of Computer-Aided Design & Graphics, 2022, 34(8): 1272-1282.

相关文章 15

[1]	刘昕晖,陈晫群,吕岩. 基于深度学习的工业故障诊断综述[J]. 吉林大学学报(工学版), 2026, 56(7): 1759-1779.
[2]	曹洁,陈志峰,王进花,陈莉. 基于扩散模型和DenseNet的少样本齿轮箱故障诊断方法[J]. 吉林大学学报(工学版), 2026, 56(7): 1787-1797.
[3]	石峰,钮鹏,樊敏. 基于Faster R-CNN算法的公路路基路面不均匀变形检测[J]. 吉林大学学报(工学版), 2026, 56(7): 1950-1957.
[4]	缪君,严杰,杜荣华,李磊,储珺. 基于双向特征融合的物体位姿估计方法[J]. 吉林大学学报(工学版), 2026, 56(2): 523-532.
[5]	周求湛,李新萌,沈皓庆子,武慧南,李媛媛,荣静,胡春华,刘萍萍. 基于注意力机制的不平衡数据的非侵入式负荷分解[J]. 吉林大学学报(工学版), 2026, 56(1): 239-246.
[6]	冯志刚,任梦媛,董冰,于明月. 基于多频带特征图和改进SqueezeNet的滚动轴承故障诊断[J]. 吉林大学学报(工学版), 2026, 56(1): 96-108.
[7]	霍震,金立生,华强,贺阳. 基于边缘特征引导的智能汽车语义分割方法[J]. 吉林大学学报(工学版), 2025, 55(9): 3032-3041.
[8]	庄珊娜,王君帅,白晶,杜京瑾,王正友. 基于三维卷积与自注意力机制的视频行人重识别[J]. 吉林大学学报(工学版), 2025, 55(7): 2409-2417.
[9]	冯志刚,王首起,于明月. 基于变分模态提取及轻量级网络的滚动轴承故障诊断[J]. 吉林大学学报(工学版), 2025, 55(6): 1883-1891.
[10]	于营,王春平,寇人可,杨博雄,王雷,赵福军,付强. 多时相高分辨率卫星遥感图像语义分割算法[J]. 吉林大学学报(工学版), 2025, 55(6): 2131-2137.
[11]	薛雅丽,俞潼安,崔闪,周李尊. 基于级联嵌套U-Net的红外小目标检测[J]. 吉林大学学报(工学版), 2025, 55(5): 1714-1721.
[12]	张河山,范梦伟,谭鑫,郑展骥,寇立明,徐进. 基于改进YOLOX的无人机航拍图像密集小目标车辆检测[J]. 吉林大学学报(工学版), 2025, 55(4): 1307-1318.
[13]	才华,王玉瑶,付强,马智勇,王伟刚,张晨洁. 基于注意力机制和特征融合的语义分割网络[J]. 吉林大学学报(工学版), 2025, 55(4): 1384-1395.
[14]	李扬,李现国,苗长云,徐晟. 基于双分支通道先验和Retinex的低照度图像增强算法[J]. 吉林大学学报(工学版), 2025, 55(3): 1028-1036.
[15]	张兰芳,李根泽,刘婷宇,余博. 局部多车影响下跟驰行为机理及建模[J]. 吉林大学学报(工学版), 2025, 55(3): 963-973.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

方法	PA/%	MIoU/%	参数量/M
CMX-B2	82.8	49.7	67
Dformer	82.7	52.0	53
SA-Gate	82.5	49.4	65
本文	83.7	52.8	45

CMFCM	CMAFM	输入	PA/%	MIoU/%
×	×	RGB	72.1	47.6
×	×	RGB-D	74.7	48.5
√	×	RGB-D	76.6	49.8
×	√	RGB-D	77.2	50.5
√	√	RGB-D	83.7	52.8