基于卷积网络注意力机制的人脸表情识别

doi:10.13229/j.cnki.jdxbgxb.20221345

摘要/Abstract

摘要：

针对表情识别时出现参数量大和识别能力弱等问题，提出一种基于卷积网络人脸表情识别方法。引入改进型残差模块，在减少参数量的同时增强对表情区域的关注；利用通道-空间注意力机制对网络提取的表情区域实现不同维度和位置上的权重分配，专注于表情关键点中细微差别特征信息；利用细节模块进一步提取深度特征信息。为得到更高准确度，引入联合损失函数延长类外距离，缩短类内距离以提高表情识别准确度。本文将此网络运用到数据集FER2013、CK+中，实验结果表明：本算法平均识别率分别为63.91%、97.98%，参数量为11.34 M。与VGG网络、残差网络等对比，该模型不仅提高了识别率，还减少了冗余参数量。

关键词: 面部表情识别, 残差模块, 通道-空间注意力机制, 细化模块

Abstract:

A convolutional network based facial expression recognition method was proposed to solve the problems of large reference number and weak recognition ability in facial expression recognition. The improved residual module was introduced to reduce the parameters and enhanced the attention to the expression area； The channel-space attention mechanism was used to assign the weights of different dimensions and positions to the expression regions extracted from the network， and the subtle feature information of the key points of expression was focused on； The refinement module was used to further extract the depth feature information. In order to obtain higher accuracy， the joint loss function was introduced to increase the out-of-class distance and reduced the in-class distance to improve the accuracy of expression recognition. The experimental results showed that the average recognition rate was 63.91% and 97.98% respectively， and the parameter was 11.34 M. Compared with VGG network and residual network， the model not only improves the recognition rate but also reduces the redundant parameters.

Key words: facial expression recognition, residual module, channel-spatial attention module, refinement module

中图分类号:

TP391

郭昕刚,程超,沈紫琪. 基于卷积网络注意力机制的人脸表情识别[J]. 吉林大学学报(工学版), 2024, 54(8): 2319-2328.

Xin-gang GUO,Chao CHENG,Zi-qi SHEN. Face expression recognition based on attention mechanism of convolution network[J]. Journal of Jilin University(Engineering and Technology Edition), 2024, 54(8): 2319-2328.

图/表 15

图1

图2

图3

图4

图 5

表1

表2

图6

图7

图8

表3

表4

表5

表6

表7

参考文献 30

1	方明, 陈文强. 结合残差网络及目标掩码的人脸微表情识别[J]. 吉林大学学报: 工学版, 2021, 51(1): 303-313.
	Fang Ming, Chen Wen-qiang. Facial micro-expression recognition based on residual error network and object mask[J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(1): 303-313.
2	Ekman P, Friesen W V. Contants across cultures in the face and emotion[J]. Journal of Personality and Social Psychology, 1971, 17(2): 124-129.
3	郑伟鹏. 基于深度学习的人脸表情识别方法研究[D]. 桂林: 广西师范大学, 2022.
	Zhen Wei-peng. Research on facial expression recognition based on deep learning[D]. Guilin: Guangxi Normal University,2022.
4	孙旭菲, 缪新颖, 毕甜甜, 等. SFSR-Age: 一种基于人脸强语义的年龄识别算法[J]. 吉林大学学报:理学版, 2024, 62(2): 347-0356.
	Sun Xu-fei, Miao Xin-ying, Bi Tian-tian, et al. SFSR-Age: an age recognition algorithm based on strong facial semantics[J]. Journal of Jilin University (Science Edition), 2024, 62(2): 347-0356.
5	葛延良, 孙笑笑, 张乔, 等. 基于循环生成对抗网络的人脸素描合成[J]. 吉林大学学报:理学版, 2022, 60(4): 897-905.
	Ge Yan-liang, Sun Xiao-xiao, Zhang Qiao, et al. Face sketch synthesis based on cycle-generative adversarial networks[J]. Journal of Jilin University (Science Edition), 2022, 60(4): 897-905.
6	Ben X Y, Ren Y, Zhang J P, et al. Video-based facial micro-expression analysis: a survey of datasets, features and algorithms[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9): 5826-5846.
7	贲晛烨, 杨明强, 张鹏, 等. 微表情自动识别综述[J]. 计算机辅助设计与图形学学报, 2014, 26(9): 1385-1395.
	Xian-yu Ben, Yang Ming-qiang, Zhang Peng, et al. Survey on automatic micro expression recognition methods[J]. Journal of Computer-Aided Design and Computer Graphics, 2014, 26(9): 1385-1395.
8	张超,卢韶芳,周付根. 眼部特征自动检测定位方法[J].吉林大学学报: 工学版,2015,45(5): 1717-1723.
	Zhang Chao, Lu Shao-fang, Zhou Fu-gen. Automatic detection localization method of eye feature[J]. Journal of Jilin University(Engineering and Technology Edition), 2015, 45(5): 1717-1723.
9	谢银成,黎曦,李天,等. 基于改进ResNet和损失函数的表情识别[J]. 自动化与仪表, 2022, 37(4): 64-69.
	Xie Yin-cheng, Li Xi, Li Tian, et al. Expression recognition based on improved ResNeT and loss function[J]. Automation and Instrumentation, 2022, 37(4): 64-69.
10	何超,侯明. 基于改进卷积神经网络的人脸表情识别方法[J]. 信息技术,2022,2022(5): 107-111.
	He Chao, Hou Ming. Facial expression recognition base on improved convolutional neural network[J]. Information Technology, 2022,2022(5) : 107-111.
11	崔子越, 皮家甜, 陈勇, 等. 结合改进 VGGNet 和Focal Loss的人脸表情识别[J]. 计算机工程与应用,2021,57(19): 171-178.
	Cui Zi-yue, Pi Jia-tian, Chen Yong, et al. Facial expression recognition combined with improved VGGNet and Focal Loss[J]. Computer Engineering and Applications, 2021, 57(19): 171-178.
12	亢洁,李佳伟,杨思力. 基于域适应卷积神经网络的人脸表情识别[J]. 计算机工程,2019,45(12): 201-206.
	Kang Jie, Li Jia-wei, Yang Si-li. Facial expression recognition based on convolution neural network with domain adaption[J]. Computer Engineering, 2019, 45(12): 201-206.
13	张波,兰艳亭,李大威,等. 基于卷积网络通道注意力的人脸表情识别[J]. 无线电工程,2022,52(1):148-153.
	Zhang Bo, Lan Yan-ting, Li Da-wei, et al. Face expression recognition based on convolution network channel attention[J]. Radio Engineering, 2022, 52(1): 148-153.
14	Jiang D H, Hu Y Z, Dai L, et al. Facial expression recognition based on attention mechanism[J]. Scientific Programming, 2021, 2021(1):1-5.
15	Sanghyun W, Jongchan P, Joonyoung L, et al. CBAM: convolutional block attention module[J]. European Conference on Computer Vision, 2018: 3-19.
16	张强, 彭骨, 薛陈斌. 基于改进图注意力网络的油井产量预测模型[J]. 吉林大学学报: 理学版, 2024, 62(4): 933-942.
	Zhang Qiang, Peng Gu, Xue Chen-bin. Oil well production prediction model based on improved graph attention network[J]. Journal of Jilin University (Science Edition), 2024, 62(4): 933-942.
17	余久方, 李中科, 陈涛. 基于分离混合注意力机制的人脸表情识别[J]. 电讯技术, 2022, 62(9): 1207-1214.
	Yu Jiu-fang, Li Zhong-ke, Chen Tao. Facial expression recognition based on separate hybird attention mechanism[J]. Telecommunication Engineering, 2022, 62(9): 1207-1214 .
18	梁华刚,王亚茹,张志伟. 基于Res-Bi-LSTM的人脸表情识别[J]. 计算机工程与应用, 2020, 56(13): 204-209.
	Liang Hua-gang, Wang Ya-ru, Zhang Zhi-wei. Facial expression recognition based on Res-Bi-LSTM[J]. Computer Engineering and Applications, 2020, 56(13): 204-209.
19	Sanghyun W, Jongchan P, Joonyoung L, et al. CBAM: Convolutional block attention module[C]// European Conference on Computer Vision, Munich, Germany, 2018: 3-19.
20	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Computer Vision and Pattern Recognition, Nevada, USA, 2016: 770-778.
21	Lucey P, Cohn J F, Kanade T, et al. The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression[C]// Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Regression-Workshops, San Francisco, USA, 2010: 94-101.
22	Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018: 7132-7141.
23	刘全明, 辛阳阳. 端到端的低质人脸图像表情识别[J]. 小型微型计算机系统, 2020, 41(3): 668-672.
	Liu Quan-ming, Xin Yang-yang. End-to-end low-quality face image expression recognition[J]. Small Microcomputer System, 2020, 41(3): 668-672.
24	Chollet F. Xception: deep learning with depth wise separable convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, USA, 2017: 1800-1807.
25	程学军, 邢萧飞. 利用改进型VGG标签学习的表情识别方法[J]. 计算机工程与设计, 2022, 43(4): 1134-1144.
	Chen Xue-jun, Xing Xiao-fei. An expression recognition method based on improved VGG tag learning[J]. Computer Engineering and Applications, 2022, 43(4):1134-1144.
26	赵家琦, 周颖玥, 王欣宇, 等. 采用支路辅助学习的人脸表情识别[J]. 计算机工程与应用, 2022, 58(23): 151-160.
	Zhao Jia-qi, Zhou Ying-yue, Wang Xin-yu, et al. Facial expression recognition using branch-assisted learning[J]. Computer Engineering and Applications. 2022, 58(23): 151-160.
27	马金峰. 基于密集连接卷积结构的人脸表情识别研究[J]. 电脑与电信, 2021, 2021(4): 1-5.
	Ma Jin-feng. Research on facial expression recognition based on dense convolution structure[J]. Computers and Telecommunications, 2021, 2021(4): 1-5.
28	高涛, 邵倩, 张亚南, 等. 基于深度残差网络的人脸表情识别研究[J]. 电子设计工程, 2020, 28 (23): 101-104.
	Gao Tao, Shao Qian, Zhang Ya-nan,et al. Research on facial expression recognition based on depth residual network[J]. Electronic Design Engineering, 2020, 28(23): 101-104.
29	刘尚旺, 刘承伟, 张爱丽. 基于深度可分离卷积神经网络的实时人脸表情和性别分类[J]. 计算机应用, 2020, 40(4): 990-995.
	Liu Shang-wang, Liu Cheng-wei, Zhang Ai-li. Real-time facial expression and gender recognition based on depthwise separable convolutional neural network[J]. 2020, 40(4): 990-995.
30	石翠萍, 谭聪, 左江, 等. 基于改进AlexNet卷积神经网络的人脸表情识别[J]. 电讯技术, 2020, 60 (9): 1005-1012.
	Shi Cui-ping, Tan Cong, Zuo Jiang, et al. Facial expression recognition based on improved AlexNet convolutional neural network[J]. Telecommunications Technology, 2020, 60(9): 1005-1012.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

预处理	IHOR	CSAM	RM	JLF	FRR2013准确率/%	CK+准确率/%
48×48	×	×	×	×	60.83	95.31
48×48	√	×	×	×	62.04	96.48
48×48	×	√	×	×	62.12	96.72
48×48	√	√	×	×	62.38	97.32
48×48	×	×	√	×	61.54	95.81
48×48	√	√	√	×	62.94	97.54
48×48	×	×	×	√	62.31	96.27
64×64	√	√	√	√	62.44	96.64
48×48	√	√	√	√	63.91	97.98

CA	SA	CL	CEL	FER2013 准确率/%	CK+ 准确率/%
√	×	√	√	62.57	96.44
×	√	√	√	62.87	96.81
√	√	√	×	63.21	97.50
√	√	×	√	62.95	97.07
√	√	√	√	63.91	97.98

识别方法	准确率/%
文献［23］	62.65
文献［22］	57.13
文献［23］	63.56
VGG16	60.40
CNN	62.08
本文	63.91

识别方法	参数量/M
VGG16	22.12
Resnet50	55.73
文献［24］	22.80
文献［25］	15.40
文献［26］	26.36
文献［27］	28.50
本文模型	11.34

真实标签	预测标签
真实标签	生气	厌恶	恐惧	高兴	中性	悲伤	惊讶
生气	1.00	0.00	0.00	0.00	0.00	0.00	0.00
厌恶	0.00	0.93	0.00	0.00	0.00	0.00	0.07
恐惧	0.00	0.00	0.96	0.04	0.00	0.00	0.00
高兴	0.00	0.00	0.00	0.89	0.00	0.11	0.00
中性	0.00	0.00	0.00	0.07	0.93	0.00	0.00
悲伤	0.00	0.00	0.05	0.00	0.07	0.88	0.00
惊讶	0.12	0.00	0.00	0.00	0.00	0.07	0.81