基于注意力机制和特征融合的语义分割网络

doi:10.13229/j.cnki.jdxbgxb.20230740

摘要/Abstract

摘要：

针对DeepLabv3+网络中的多尺度目标分割错误、多尺度特征图及不同阶段特征图之间关联性差的问题，提出在DeepLabv3+基础上引入全局上下文注意力模块、级联自适应尺度感知模块及注意力优化融合模块。将全局上下文注意力模块嵌入骨干网络特征提取的初始阶段，获取丰富的上下文信息；级联自适应尺度感知模块可建模多尺度特征之间的依赖性，使其更加关注目标特征；注意力优化融合模块通过多条支路融合多层特征，以此提高解码时像素的连续性。改进网络在Cityscapes数据集以及PASCAL VOC2012增强数据集上进行验证测试，实验结果表明：该网络能弥补DeepLabv3+的不足，且平均交并比分别达到76.2%、78.7%。

关键词: 语义分割, 多尺度特征, 上下文信息, 注意力机制, 特征融合

Abstract:

To address the issues of multi-scale object segmentation errors， poor correlation between multi-scale feature maps and feature maps at different stages in the DeepLabv3+ network， the following modules are proposed to incorporate，including a global context attention module， a cascade adaptive Scale awareness module， and an attention optimized fusion module. The global context attention module is embedded in the initial stage of the backbone network for feature extraction， allowing it to capture rich contextual information. The cascade adaptive scale awareness module models the dependencies between multi-scale features， enabling a stronger focus on the features relevant to the target. The attention optimized fusion module merges multiple layers of features through multiple pathways to enhance pixel continuity during decoding. The improved network is validated on the CityScapes dataset and PASCAL VOC2012 augmented dataset， and the experimental results demonstrate its ability to overcome the limitations of DeepLabv3+. Furthermore， the mean intersection over union reaches 76.2% and 78.7% respectively.

Key words: semantic segmentation, multi-scale features, contextual information, attention mechanism, feature fusion

中图分类号:

TP391.4

才华,王玉瑶,付强,马智勇,王伟刚,张晨洁. 基于注意力机制和特征融合的语义分割网络[J]. 吉林大学学报(工学版), 2025, 55(4): 1384-1395.

Hua CAI,Yu-yao WANG,Qiang FU,Zhi-yong MA,Wei-gang WANG,Chen-jie ZHANG. Semantic segmentation network based on attention mechanism and feature fusion[J]. Journal of Jilin University(Engineering and Technology Edition), 2025, 55(4): 1384-1395.

图/表 14

图1

图2

图3

图4

图5

图6

表1

不同骨干网络在Cityscapes验证集上的比较"

模型

骨干网络

M I o U

Baseline

+GCAM+CASAM+AOFM1，2

MobileNetV2

72.1

72.5

Baseline

+GCAM+CASAM+AOFM1，2

ResNet18

73.2

74.3

Baseline

+GCAM+CASAM+AOFM1，2

ResNet50

73.8

75.6

Baseline

+GCAM+CASAM+AOFM1，2

ResNet101

73.9

76.2

表1

表2

改进网络在Cityscapes验证集上的消融实验"

DeepLabv3+	GCAM	CASAM	AOFM1	AOFM2	参数量/M	$M I o U$ /%
√					52.7	73.9
√	√				58.2	74.2
√	√	√			64.3	75.8
√	√	√	√		66.2	76.1
√	√	√	√	√	67.5	76.2

表2

图7

表3

不同算法在Cityscapes验证集上的对比"

算法	骨干网络	参数量/M	$M I o U$ /%
SegNet	VGG16	50.85	66.7
CCNet	ResNet50	60.8	71.4
UperNet	ResNet101	86.4	72.5
OCRNet	HRNet	72.8	73.4
DANet	ResNet50	58.9	73.9
Segformer	MiT	101.2	75.9
SETR-MLA	T-Small	311.8	76.1
DeepLabv3+	ResNet101	52.7	74.2
本文	ResNet101	67.5	76.2

表3

表4

图8

表5

不同算法在PASCAL VOC2012验证集的对比"

算法	骨干网络	参数量/M	Flops/G	$M I o U$ /%
SegNet	VGG16	20.32	22.3	70.1
CCNet	ResNet50	47.2	60.1	71.5
UperNet	ResNet101	53.4	64.5	72.4
OCRNet	ResNet101	69.4	68.4	73.7
DANet SETR-MLA	ResNet50 T-Small	10.34 180.62	51.4 65.2	76.3 78.3
Segformer	MiT	100.4	63.9	78.6
DeepLabv3+	ResNet101	32.7	56.6	72.9
本文	ResNet101	40.2	62.3	78.7

表5

图9

参考文献 28

[1]	Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for biomedical image segmentation[C]∥Medical Image Computing and Computer-Assisted Intervention-MICCAI: The 18th International Conference, Munich, Germany, 2015: 234-241.
[2]	Chen J, Lu Y, Yu Q, et al. Transunet: transformers make strong encoders for medical image segmentation[J/OL]. [2023-07-02].arXiv preprint arXiv: 2102. 04306v1.
[3]	Zhao T Y, Xu J D, Chen R, et al. Remote sensing image segmentation based on the fuzzy deep convolutional neural network[J]. International Journal of Remote Sensing, 2021, 42(16): 6264-6283.
[4]	Yuan X H, Shi J F, Gu L C. A review of deep learning methods for semantic segmentation of remote sensing imagery[J]. Expert Systems with Applications, 2021, 169: No.114417.
[5]	Xu Z Y, Zhang W, Zhang T X, et al. Efficient transformer for remote sensing image segmentation[J]. Remote Sensing, 2021, 13(18): No.3585.
[6]	Badrinarayanan V, Kendall A, Cipolla R. Segnet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.
[7]	Yu C, Gao C, Wang J, et al. Bisenet v2: bilateral network with guided aggregation for real-time semantic segmentation[J]. International Journal of Computer Vision, 2021, 129: 3051-3068.
[8]	Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA,2015: 3431-3440.
[9]	Chen L C, Papandreou G Kokkinos I, et al. Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848.
[10]	Chen L C, Papandreou G, Schroff F, et al. Rethinking atrous convolution for semantic image segmentation[J/OL].[2023-07-03]. arXiv preprint arXiv: 1706. 05587v3.
[11]	Chen L C, Zhu Y, Papandreou G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]∥Proceedings of the European conference on computer vision (ECCV),Munich, Germany,2018: 833-851.
[12]	Wang J, Sun K, Cheng T, et al. Deep high-resolution representation learning for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(10): 3349-3364.
[13]	Liu Z, Lin Y, Cao Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]∥Proceedings of the IEEE/CVF International Conference on Computer Vision,Montreal, Canada, 2021: 10012-10022.
[14]	Wang W, Xie E, Li X, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions[C]∥Proceedings of the IEEE/CVF International Conference on Computer Vision,Montreal, Canada, 2021: 568-578.
[15]	Zheng S, Lu J, Zhao H, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA,2021: 6881-6890.
[16]	Xie E, Wang W, Yu Z, et al. SegFormer: simple and efficient design for semantic segmentation with transformers[J]. Advances in Neural Information Processing Systems, 2021, 34: 12077-12090.
[17]	Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[J/OL]. [2023-07-04].arXiv preprint arXiv: 2010. 11929v2.
[18]	Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 2881-2890.
[19]	Hou Q, Zhang L, Cheng M M, et al. Strip pooling: rethinking spatial pooling for scene parsing[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Seattle, USA, 2020: 4003-4012.
[20]	Peng C, Zhang X, Yu G, et al. Large kernel matters-improve semantic segmentation by global convolutional network[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 4353-4361.
[21]	Ding X, Zhang X, Han J, et al. Scaling up your kernels to 31×31: revisiting large kernel design in CNNs[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,New Orleans, USA, 2022: 11963-11975.
[22]	Guo M H, Lu C Z, Liu Z N, et al. Visual attention network[J/OL]. [2023-07-04].arXiv preprint arXiv:.
[23]	Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 7132-7141.
[24]	Wang Q, Wu B, Zhu P, et al. ECA-Net: efficient channel attention for deep convolutional neural networks[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Seattle, USA, 2020: 11534-11542.
[25]	Fu J, Liu J, Tian H, et al. Dual attention network for scene segmentation[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 3146-3154.
[26]	Cordts M, Omran M, Ramos S, et al. The cityscapes dataset for semantic urban scene understanding[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Las Vegas, USA, 2016: 3213-3223.
[27]	Everingham M, Eslami S M A, Van Gool L, et al. The pascal visual object classes challenge: a retrospective[J]. International Journal of Computer Vision, 2015, 111: 98-136.
[28]	王雪, 李占山, 吕颖达. 基于多尺度感知和语义适配的医学图像分割算法[J]. 吉林大学学报: 工学版, 2022, 52(3): 640-647.
	Wang Xue, Li Zhan-shan, Ying-da Lyu. Medical image segmentation algorithm based on multi-scale perception and semantic adaptation [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 640-647.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

算法	道路	人行道	建筑物	墙体	围栏	交通灯	交通标识	电线杆	植被	地形	天空	行人	骑行者	汽车	卡车	公交车	火车	摩托车	自行车	MIoU/%
SegNet	89.8	75.4	81.7	38.7	58.7	60.3	57.9	59.8	70.1	56.7	80.6	72.4	58.3	89.6	62.3	68.4	67.6	51.3	67.9	66.7
CCNet	91.1	78.8	86.3	50.9	60.2	61.5	66.8	61.4	73.2	58.0	84.4	75.2	59.1	79.2	73.7	83.1	76.3	66.2	71.2	71.4
UperNet	92.4	79.6	87.4	51.2	58.1	62.9	67.4	63.5	73.4	59.1	86.6	76.3	60.3	85.6	77.1	83.1	77.5	64.6	71.4	72.5
OCRNet	93.3	79.4	88.1	51.8	59.6	63.7	68.3	63.7	74.2	58.9	87.3	77.8	61.4	86.9	78.4	84.2	78.6	66.7	72.3	73.4
DANet	93.6	80.1	88.3	47.4	58.9	64.5	69.1	64.5	74.1	60.7	87.6	78.3	60.6	87.9	79.9	86.7	79.2	68.6	73.3	73.9
Segformer	93.8	80.5	88.9	55.4	61.9	66.1	72.6	66.0	78.4	61.3	87.9	79.4	62.7	90.6	82.2	86.9	81.6	69.5	76.4	75.9
SETR-MLA	94.2	80.3	89.6	55.9	62.3	66.4	71.8	64.3	77.5	62.8	88.7	80.2	62.9	90.4	82.6	86.7	81.9	69.4	77.1	76.1
DeepLabv3+	93.8	80.2	88.6	53.2	59.6	64.5	69.5	63.4	76.1	60.4	87.3	78.5	60.3	89.1	79.3	85.6	79.9	67.6	73.7	74.2
本文	94.3	81.1	89.4	56.6	62.6	66.7	71.2	66.7	77.2	62.7	88.2	80.7	62.8	90.8	82.7	87.2	82.3	69.5	75.2	76.2