Abstract:

A multimodal semantic segmentation model was proposed for indoor scene perception tasks. Existing multimodal fusion methods mainly focus on a single modality， such as RGB or depth image attention， while lacking effective utilization of attention in the joint feature space after modality fusion. Additionally， many networks suffer from a significant drop in accuracy when the number of parameters is reduced， due to their large parameter sizes. The RGB and depth features are extracted using ConvNeXt and MiT， respectively. A Cross-modal Feature Correction Module （CMFCM） is introduced to enhance spatial and channel features， facilitating interaction and correction between modalities. Additionally， a Cross-modal Attention Fusion Module （CMAFM） is designed to apply attention in the joint feature space， enabling effective multi-scale feature fusion and maximizing the utilization of segmentation information from both modalities. Depthwise separable convolution is also employed to significantly reduce computational costs. Finally， experiments on the NYU-Depth V2 dataset show that， compared with CMX， Dformer， and SA-Gate models， the proposed method improves segmentation accuracy by 1% while reducing the number of parameters by 15.1%， significantly lowering computational complexity.

Key words: semantic segmentation, multimodal fusion, indoor perception, attention mechanism