基于语义耦合和身份一致性的跨模态行人重识别方法

doi:10.13229/j.cnki.jdxbgxb20210481

摘要/Abstract

摘要：

针对跨模态行人重识别面临的较大跨模态差异和类内变化的问题，提出了一种基于语义耦合和身份一致性的跨模态行人重识别方法。在语义层面，通过双向耦合不同模态的语义特征，实现不同模态间语义的交互融合，有效缓解了跨模态差异；在行人身份层面，通过优化跨模态三元组损失和身份损失，实现类内身份信息一致性，有效缓解了类内变化问题。实验结果表明，本文算法能够有效提升跨模态行人重识别精度，与基线方法相比，Top-1和mAP指标精度提升了10%以上。

关键词: 计算机应用, 跨模态行人重识别, 深度学习, 语义耦合, 身份一致性约束

Abstract:

To solve the problem of the large inter-modality discrepancy and the intra-class variations in cross modality person re-Identification （CM-Reid）, a novel CM-Reid framework based on semantic coupling and identity-consistence constraint was proposed. In the semantic level， the semantic representations bi-directionally and fuse the semantic information between different modalities were coupled to alleviate the inter-modality discrepancy. In the identity level， the cross-modality triplet loss and identity loss to maintain the identity consistence were optimized to alleviate the intra-class variations. The experimental results show that the proposed method can effectively improve the performance of CM-Reid. Compared with the baseline method, the accuracy of Top-1 and mAP indicators is improved by more than 10%.

Key words: computer application, cross-modality person re-identification, deep learning, semantic coupling, identity-consistence constrain

中图分类号:

TP391

侯春萍,杨庆元,黄美艳,王致芃. 基于语义耦合和身份一致性的跨模态行人重识别方法[J]. 吉林大学学报(工学版), 2022, 52(12): 2954-2963.

Chun-ping HOU,Qing-yuan YANG,Mei-yan HUANG,Zhi-peng WANG. Cross⁃modality person re⁃identification based on semantic coupling and identity⁃consistence constraint[J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(12): 2954-2963.

图/表 8

图1

图2

表1

表2

表3

表4

图3

图4

参考文献 25

1	Ye Mang, Shen Jian-bing, Lin Gao-jie, et al. Deep learning for person re-identification: a survey and outlook[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(6): 2872-2893.
2	Zhong Zhun, Zheng Liang, Luo Zhi-ming, et al. Learning to adapt invariance in memory for person re-identification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(8): 2723-2738.
3	Qian Xue-lin, Fu Yan-wei, Tao Xiang, et al. Leader-based multi-scale attention deep architecture for person re-identification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 42(2): 371-385.
4	马淼, 王云涛, 潘海鹏. 基于注意力机制和空间几何约束的行人重识别方法[J]. 吉林大学学报: 工学版, 2022, 52(5): 1079-1087.
	Ma Miao, Wang Yun-tao, Pan Hai-peng. Person re-identification method based on attention mechanism and spatial geometric constraint[J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(5): 1079-1087.
5	Chen Guang-yi, Lu Ji-wen, Yang Ming, et al. Learning recurrent 3D attention for video-based person re-identification[J]. IEEE Transactions on Image Processing, 2020, 29: 6963-6976.
6	Wang K, Ding C, Maybank S J, et al. CDPM: convolutional deformable part models for semantically aligned person re-identification[J]. IEEE Transactions on Image Processing, 2019, 29: 3416-3428.
7	Wu A, Zheng W S, Yu H X, et al. RGB-infrared cross-modality person re-identification[C]∥Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 2017: 5380-5389.
8	Ye Mang, Wang Zheng, Lan Xiang-yuan, et al. Visible thermal person re-identification via dual-constrained top-ranking[C]∥International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 2018:1092-1099.
9	Hao Yi, Wang Nan-nan, Li Jie, et al. HSME: hypersphere manifold embedding for visible thermal person re-identification[C]∥Proceedings of the AAAI Conference on Artificial Intelligence, Hawaii, USA,2019: 8385-8392.
10	Wang Guan-an, Zhang Tian-zhu, Cheng Jian, et al. RGB-infrared cross-modality person re-identification via joint pixel and feature alignment[C]∥Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 2019: 3623-3632.
11	Wang Zhi-xiang, Wang Zheng, Zheng Yin-qiang, et al. Learning to reduce dual-level discrepancy for infrared-visible person re-identification[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019: 618-626.
12	Deng J, Dong W, Socher R, et al. ImageNet: a large-scale hierarchical image database[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Miami, USA, 2009: 248-255.
13	He Kai-ming, Zhang Xiang-yu, Ren Shao-qing, et al. Deep residual learning for image recognition[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770-778.
14	Hermans A, Beyer L, Leibe B. In defense of the triplet loss for person re-identification[DB/OL]. [2020-12-03].
15	Nguyen D T, Hong H G, Kim K W, et al. Person recognition system based on a combination of body images from visible light and thermal cameras[J]. Sensors, 2017, 17(3): No.605.
16	Kansal K, Subramanyam A V, Wang Z, et al. SDL: spectrum-disentangled representation learning for visible-infrared person re-identification[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(10): 3422-3432.
17	Feng Zhan-xiang, Lai Jian-hua, Xie Xiao-hua. Learning modality-specific representations for visible-infrared person re-identification[J]. IEEE Transactions on Image Processing, 2019, 29: 579-590.
18	Ye Mang, Lan Xiang-yuan, Leng Qing-ming, et al. Cross-modality person re-identification via modality-aware collaborative ensemble learning[J]. IEEE Transactions on Image Processing, 2020, 29: 9387-9399.
19	Ye Mang, Lan Xiang-hua, Wang Zheng, et al. Bi-directional center-constrained top-ranking for visible thermal person re-identification[J]. IEEE Transactions on Information Forensics and Security, 2019, 15: 407-419.
20	Ye Han-rong, Liu Hong, Meng Fan-yang, et al. Bi-directional exponential angular triplet loss for RGB-infrared person re-identification[J]. IEEE Transactions on Image Processing, 2021, 30: 1583-1595.
21	Wu An-cong, Zheng Wei-shi, Gong Shao-gang, et al. RGB-IR person re-identification by cross-modality similarity preservation[J]. International Journal of Computer Vision, 2020, 128(6): 1765-1785.
22	Dai P Y, Ji R R, Wang H B, et al. Cross-modality person re-identification with generative adversarial training[C]∥Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, Stockholm, Sweden: 2018: 677-683.
23	Wang G A, Zhang T, Yang Y, et al. Cross-modality paired-images generation for RGB-infrared person re-identification[C]∥Proceedings of the AAAI Conference on Artificial Intelligence, New York, USA, 2020: 12144-12151.
24	胡静,陶洋.基于RPCA的群稀疏表示人脸识别方法[J]. 重庆邮电大学学报: 自然科学版, 2020, 32(3): 459-468.
	Hu Jing, Tao Yang. Group sparse representation face recognition method based on RPCA[J]. Journal of Chongqing University of Posts and Telecommunications (Natural Science Edition), 2020, 32(3): 459-468.
25	李新春,马红艳,林森.基于局部邻域四值模式的掌纹掌脉融合识别[J].重庆邮电大学学报: 自然科学版, 2020, 32(4): 630-638.
	Li Xin-chun, Ma Hong-yan, Lin Sen. Palmprint and palm vein fusion recognition based on local neighbor quaternary pattern[J]. Journal of Chongqing University of Posts and Telecommunications (Natural Science Edition), 2021, 47(5): 285-291, 300.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

序号	方法	All Search		Indoor Search
序号	方法	Top-1	mAP	Top-1	mAP
1	Baseline	45.81	44.85	47.06	46.13
2	Baseline+SCM_block1	46.01	45.15	47.98	46.05
3	Baseline+SCM_block2	48.97	48.00	50.10	49.25
4	Baseline+SCM_block3	54.10	52.61	55.83	52.96
5	Baseline+SCM_block4	57.85	55.33	58.15	56.19

方法	All Search		Indoor Search
方法	Top-1	mAP	Top-1	mAP
1	55.92	54.20	56.85	54.92
2	57.85	55.33	58.15	56.19

方法	All Search				Indoor Search
方法	Top-1	Top-10	Top-20	mAP	Top-1	Top-10	Top-20	mAP
One-stream^［7］	12.04	49.68	66.74	13.67	16.94	63.55	82.10	22.95
Two-stream^［7］	11.65	47.99	65.50	12.85	15.60	61.18	81.02	21.49
Zero-Padding^［7］	14.80	54.12	71.33	15.95	20.58	68.38	85.79	26.92
SDL^［16］	28.12	70.23	83.67	29.01	32.56	80.45	90.67	39.56
MSR^［17］	37.35	83.40	93.34	38.11	39.64	89.29	97.66	50.88
MACE^［18］	51.64	87.25	94.44	50.11	57.35	93.02	97.47	64.79
HSM^［9］	20.68	32.74	77.95	23.21	-	-	-	-
BDTR^［19］	27.32	66.96	81.07	27.32	31.92	77.18	89.28	41.86
eBDTR^［19］	27.82	67.34	81.34	28.42	32.46	77.42	89.62	42.46
expAT^［20］	38.57	76.64	86.39	38.61	44.71	69.82	77.87	32.20
CMSP^［21］	43.56	86.25	-	44.98	48.62	89.50	-	57.50
cmGAN^［22］	26.97	67.51	80.56	31.49	31.63	77.23	89.18	42.19
D2RL^［11］	28.90	70.60	82.40	29.20	-	-	-	-
JSIA^［23］	38.10	80.70	89.90	36.90	43.80	86.20	94.20	52.90
AliGAN^［10］	42.40	85.00	93.70	40.70	45.90	87.60	94.40	54.30
本文	57.85	89.25	94.45	55.33	58.15	90.02	95.63	56.19

方法	RGB→Thermal				Thermal→RGB
方法	Top-1	Top-10	Top-20	mAP	Top-1	Top-10	Top-20	mAP
Zero-Padding^［7］	17.75	34.21	44.35	18.90	16.63	34.68	44.25	17.82
SDL^［16］	26.47	51.34	61.22	23.58	25.74	50.23	59.66	22.89
MSR^［17］	48.43	70.32	79.95	48.67	-	-	-	-
MACE^［18］	72.37	88.40	93.59	69.09	72.12	88.07	93.07	68.57
BDTR^［19］	33.56	58.61	67.43	32.76	32.92	58.46	68.43	31.96
eBDTR^［19］	34.62	58.96	68.72	33.46	34.21	58.74	68.64	32.49
HSME^［9］	50.85	73.36	81.66	47.00	50.15	72.40	81.07	46.16
expAT^［20］	67.45	-	-	66.51	66.48	-	-	67.31
CMSP^［21］	65.07	83.71	-	64.50	-	-	-	-
D2RL^［11］	43.40	66.10	76.30	44.10	-	-	-	-
JSIA^［23］	48.50	-	-	48.90	-	-	-	-
AliGAN^［10］	57.90	-	-	53.60	56.30	-	-	53.40
本文	75.12	90.56	96.09	74.51	74.96	91.02	95.76	76.10

[1]	祁贤雨,王巍,王琳,赵玉飞,董彦鹏. 基于物体语义栅格地图的语义拓扑地图构建方法[J]. 吉林大学学报(工学版), 2023, 53(2): 569-575.
[2]	时小虎,吴佳琦,吴春国,程石,翁小辉,常志勇. 基于残差网络的弯道增强车道线检测方法[J]. 吉林大学学报(工学版), 2023, 53(2): 584-592.
[3]	郭鹏,赵文超,雷坤. 基于改进Jaya算法的双资源约束柔性作业车间调度[J]. 吉林大学学报(工学版), 2023, 53(2): 480-487.
[4]	刘近贞,高国辉,熊慧. 用于脑组织分割的多尺度注意网络[J]. 吉林大学学报(工学版), 2023, 53(2): 576-583.
[5]	高金武,贾志桓,王向阳,邢浩. 基于PSO-LSTM的质子交换膜燃料电池退化趋势预测[J]. 吉林大学学报(工学版), 2022, 52(9): 2192-2202.
[6]	周丰丰,朱海洋. 基于三段式特征选择策略的脑电情感识别算法SEE[J]. 吉林大学学报(工学版), 2022, 52(8): 1834-1841.
[7]	申铉京,张雪峰,王玉,金玉波. 像素级卷积神经网络多聚焦图像融合算法[J]. 吉林大学学报(工学版), 2022, 52(8): 1857-1864.
[8]	曲福恒,丁天雨,陆洋,杨勇,胡雅婷. 基于邻域相似性的图像码字快速搜索算法[J]. 吉林大学学报(工学版), 2022, 52(8): 1865-1871.
[9]	白天,徐明蔚,刘思铭,张佶安,王喆. 基于深度神经网络的诉辩文本争议焦点识别[J]. 吉林大学学报(工学版), 2022, 52(8): 1872-1880.
[10]	李晓英,杨名,全睿,谭保华. 基于深度学习的不均衡文本分类方法[J]. 吉林大学学报(工学版), 2022, 52(8): 1889-1895.
[11]	赵宏伟,张健荣,朱隽平,李海. 基于对比自监督学习的图像分类框架[J]. 吉林大学学报(工学版), 2022, 52(8): 1850-1856.
[12]	秦贵和,黄俊锋,孙铭会. 基于双手键盘的虚拟现实文本输入[J]. 吉林大学学报(工学版), 2022, 52(8): 1881-1888.
[13]	胡丹,孟新. 基于时变网格的对地观测卫星搜索海上船舶方法[J]. 吉林大学学报(工学版), 2022, 52(8): 1896-1903.
[14]	王军,徐彦惠,李莉. 低能耗支持完整性验证的数据融合隐私保护方法[J]. 吉林大学学报(工学版), 2022, 52(7): 1657-1665.
[15]	周丰丰,张亦弛. 基于稀疏自编码器的无监督特征工程算法BioSAE[J]. 吉林大学学报(工学版), 2022, 52(7): 1645-1656.