双向特征金字塔全景分割网络

doi:10.13229/j.cnki.jdxbgxb20200851

Abstract

Abstract:

To overcome the deficiency of traditional feature pyramid network （FPN） used in the field of panoptic segmentation， a Two-way FPN for panoptic segmentation （T-FPN） is proposed. The network solves the contradiction between the unified feature output caused by the one-way transmission of FPN and dual-threaded task feature requirements of panoptic segmentation. Based on the analysis of the difference between the foreground and background of the image， the two-way transmission paths are constructed according to the feature requirements of the foreground and background segmentation tasks， the top-down path uses upsampling to strengthen foreground feature， and the bottom-up path enhances background feature with dilated convolution. The two-way network is able to extract foreground feature and background feature simultaneously， which is helpful to create a dynamic balance of the foreground and background segmentation accuracy， thereby improving the quality of panoptic segmentation. The experimental results on the MS COCO and Cityscapes datasets show that our T-FPN is superior to the existing similar methods in terms of segmentation accuracy. Compared with UPSNet that uses traditional feature pyramid network， the PQ value of T-FPN is increased by 0.47%.

Key words: computer application, panoptic segmentation, feature pyramid network（FPN）, foreground, background, dilated convolution

CLC Number:

TP391.41

Lin MAO,Feng-zhi REN,Da-wei YANG,Ru-bo ZHANG. Two⁃way feature pyramid network for panoptic segmentation[J].Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 657-665.

Figures/Tables 8

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Table 2

Fig.6

References 24

1	Li Y, Chen X, Zhu Z, et al. Attention-guided unified network for panoptic segmentation[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019: 7019-7028.
2	Kirillov A, Girshick R, He K, et al. Panoptic feature pyramid networks[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019: 6392-6401.
3	Lin T, Dollár P, Girshick R, et al. Feature pyramid networks for object detection[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017: 936-944.
4	Liu H Y, Peng C, Yu C Q, et al. An end-to-end network for panoptic segmentation[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019: 6165-6174.
5	Xiong Y W, Liao R J, Zhao H S, et al. UPSNet: a unified panoptic segmentation network[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019: 8810-8818.
6	Li J, Raventos A, Bhargava A, et al. Learning to fuse things and stuff[J/OL]. [2020-10-22].
7	Badrinarayanan V, Kendall A, Cipolla R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.
8	徐谦, 李颖, 王刚. 基于深度学习图像语义分割的机器人环境感知[J]. 吉林大学学报:工学版, 2019, 49(1): 248-260.
	Xu Qian, Li Ying, Wang Gang. Robotic environment sensing based on semantic segmentation by deep learning[J]. Journal of Jilin University (Engineering and Technology Edition), 2019, 49(1): 248-260.
9	Fan L, Kong H F, Wang W C, et al. Semantic segmentation with global encoding and dilated decoder in street scenes[J]. IEEE Access, 2018, 6: 50333-50343.
10	Ding H H, Jiang X D, Shuai B, et al. Semantic segmentation with context encoding and multi-path decoding[J]. IEEE Transactions on Image Processing, 2020, 29: 3520-3533.
11	Yang T J, Collins M D, Zhu Y, et al. DeeperLab: single-shot image parser[J/OL]. [2020-10-23].
12	Cheng B, Collins M D, Zhu Y K, et al. Panoptic-DeepLab: a simple, strong, and fast baseline for bottom-up panoptic segmentation[J/OL]. [2020-10-23].
13	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Nevada, USA, 2016: 770-778.
14	张涛, 张乐. 一种基于多尺度特征融合的目标检测算法[J]. 激光与光电子学进展, 2021, 58(2): 294-300.
	Zhang Tao, Zhang Le. An object detection algorithm based on multi-scale feature fusion[J]. Laser & Optoelectronics Progress, 2021, 58(2): 294-300.
15	Tian Z, Shen C H, Chen H, et al. FCOS: a simple and strong anchor-free object detector[J/OL]. [2020-10-22].
16	汤润发, 宋慧慧, 张开华, 等. 特征注意金字塔调制网络的视频目标分割[J]. 中国图象图形学报, 2019, 24(8): 1349-1357.
	Tang Run-fa, Song Hui-hui, Zhang Kai-hua, et al. Video object segmentation via feature attention pyramid modulating network[J]. Journal of Image and Graphics, 2019, 24(8): 1349-1357.
17	Yu F, Koltun V. Multi-Scale context aggregation by dilated convolutions[J/OL]. [2020-10-26].
18	He K, Gkioxari G, Dollar P, et al. Mask R-CNN[C]∥2017 IEEE International Conference on Computer Vision, Venice, Italy, 2017: 2980-2988.
19	潘晓英, 魏德, 赵逸喆. 基于Mask R-CNN和上下文卷积神经网络的肺结节检测[J/OL]. [2021-06-18].
20	Dai Ji-feng, Qi Hao-zhi, Xiong Yu-wen, et al. Deformable convolutional networks[C]∥IEEE International Conference on Computer Vision, Venice, Italy, 2017: 764-773.
21	Lin T Y, Maire M, Belongie S, et al. Microsoft COCO: common objects in context[C]∥European Conference on Computer Vision, Zurich, Switzerland, 2014: 740-755.
22	Cordts M, Omran M, Ranos S, et al. The cityscapes dataset for semantic urban scene understanding[C]∥IEEE Conference on Computer Vision and Pattern Recognition Las Vegas, Nevada, USA, 2016: 3213-3223.
23	杨丽娟, 李利. 基于双线性插值的内容感知图像缩放算法仿真[J]. 计算机仿真, 2019, 36(12): 244-248.
	Yang Li-juan, Li Li. Simulation of content-aware image scaling algorithm based on bilinear interpolation[J]. Computer Simulation, 2019, 36(12): 244-248.
24	Kirillov A, He K, Girshick R, et al. Panoptic segmentation[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019: 9396-9405.

Related Articles 15

[1]	Xue WANG,Zhan-shan LI,Ying-da LYU. Medical image segmentation based on multi⁃scale context⁃aware and semantic adaptor [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 640-647.
[2]	Ji-hong OUYANG,Ze-qi GUO,Si-guang LIU. Dual⁃branch hybrid attention decision net for diabetic retinopathy classification [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 648-656.
[3]	Su-ming KANG,Ye-e ZHANG. Hadoop⁃based local timing link prediction algorithm across social networks [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 626-632.
[4]	You QU,Wen-hui LI. Single-stage rotated object detection network based on anchor transformation [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(1): 162-173.
[5]	Hong-wei ZHAO,Dong-sheng HUO,Jie WANG,Xiao-ning LI. Image classification of insect pests based on saliency detection [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2174-2181.
[6]	Zhou-zhou LIU,Qian-yun ZHANG,Xin-hua MA,Han PENG. Compressed sensing signal reconstruction based on optimized discrete differential evolution algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2246-2252.
[7]	Sheng-sheng WANG,Jing-yu CHEN,Yi-nan LU. COVID⁃19 chest CT image segmentation based on federated learning and blockchain [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2164-2173.
[8]	Dong-ming SUN,Liang HU,Yong-heng XING,Feng WANG. Text fusion based internet of things service recommendation for trigger⁃action programming pattern [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2182-2189.
[9]	Jun-cong LIN,Jun LEI,Meng CHEN,Shi-hui GUO,Xing GAO,Ming-hong LIAO. Real⁃time camera planning for dynamic multiple targets considering cinematographic visual properties [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2154-2163.
[10]	Li-li REN,Zhi-jun WANG,Dong-mei YAN. Improved multi⁃verse algorithm with combined slime mould foraging behavior [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2190-2197.
[11]	Yin-di YAO,Jun-jin HE,Yang-li LI,Dang-yuan XIE,Ying LI. ET₀ simulation of self⁃constructed improved whale optimized BP neural network [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(5): 1798-1807.
[12]	Hong-wei ZHAO,Zi-jian ZHANG,Jiao LI,Yuan ZHANG,Huang-shui HU,Xue-bai ZANG. Bi⁃direction segmented anti⁃collision algorithm based on query tree [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(5): 1830-1837.
[13]	Jie CAO,Xue QU,Xiao-xu LI. Few⁃shot image classification method based on sliding feature vectors [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(5): 1785-1791.
[14]	Xiao-xue SUN,Hui ZHONG,Hai-peng CHEN. Statistical analysis system for students' examination results based on decision tree classification technology [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(5): 1866-1872.
[15]	Meng-su ZHANG,Chun-tian LIU,Xi-jin LI,Yong-ping HUANG. Design of fuzzy comprehensive evaluation system for performance appraisal based on K⁃means clustering algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(5): 1851-1856.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

Method	Backbone	PQ	PQ^Th	PQ^St	SQ	RQ
DeeperLab	Xception71	34.3	37.5	29.6	77.1	43.1
AUNet	ResNet101+FPN	39.6	49.1	25.2	-	-
TASCNet	ResNet101+FPN	40.7	47.0	31.0	78.5	50.1
Panoptic FPN	ResNet50+FPN	40.9	48.3	29.7	-	-
OANet	ResNet50+FPN	41.3	50.4	27.7	-	-
UPSNet	ResNet50+FPN	42.5	48.5	33.4	78.0	52.4
T-FPN	ResNet50+T-FPN	42.7	48.7	33.6	78.6	52.7

Method	Backbone	PQ	PQ^Th	PQ^St	SQ	RQ
DeeperLab	Xception71	56.5	-	-	-	-
Panoptic FPN	ResNet101+FPN	58.1	52.0	62.5	-	-
AUNet	ResNet101+FPN	59.0	54.8	62.1	-	-
TASCNet	ResNet50+FPN	59.2	56.0	61.5	-	-
UPSNet	ResNet50+FPN	59.3	54.6	62.7	79.7	73.0
T-FPN	ResNet50+T-FPN	59.3	54.5	62.8	79.3	73.2

Two⁃way feature pyramid network for panoptic segmentation

RICH HTML

PDF (PC)