结合残差网络及目标掩膜的人脸微表情识别

doi:10.13229/j.cnki.jdxbgxb20190939

摘要/Abstract

摘要：

微表情是一种能反映隐藏在人内心最真实情感状态的面部特征，现有的微表情识别技术经常会提取一些与微表情无关的运动特征信息干扰情感的真实性识别。为解决上述问题，提出一种结合残差网络及目标掩膜的特征提取方法。本文首先对原始的微表情序列进行一系列预处理，定位人眼的关键区域，对眼部区域进行图像掩膜，进而减少眨眼动作对微表情特征提取造成的干扰，然后使用欧拉视频放大算法对人脸微表情变化的关键区域进行放大，使得微表情变化更为明显，最后通过3D ResNet网络对连续的微表情序列进行训练和识别。本文方法在CASME II和SMIC数据集上进行了测试，准确率分别达到77.3%和72.4%，与最新方法DSSN、CNN+LSTM等相比，准确率至少提高5%。

关键词: 计算机应用, 微表情, 3D ResNet, 目标掩膜, 欧拉视频放大, 情感识别

Abstract:

Micro-expression is a kind of facial feature that can reflect the most real emotional state hidden in human heart. Existing micro-expression recognition technology often extracts a lot of feature information unrelated to micro-expression in feature extraction. In order to solve the above problem， this paper proposes a feature extraction method of specific region of face micro-expression based on deep learning. Firstly， we preprocess the original micro-expression sequence to locate the key regions of human eyes， mask the eye regions， and reduce the blinking action to extract micro-expression features. Then， Euler video magnification algorithm is used to magnify the key areas of facial micro-expression change， which makes the part of micro-expression change more obvious. Finally， the continuous micro-expression sequence is trained and recognized by 3D ResNet network. The proposed method is tested on CASME II and SMIC datasets， and he experimental results show that the accuracies are 77.3% and 72.4% respectively. Compared with the latest methods DSSN， CNN+LSTM， the accuracy is improved at least 5%.

Key words: computer application, micro-expression, 3D ResNet, object mask, Eulerian video magnification, emotion recognition

中图分类号:

TP391.4

方明,陈文强. 结合残差网络及目标掩膜的人脸微表情识别[J]. 吉林大学学报(工学版), 2021, 51(1): 303-313.

Ming FANG,Wen-qiang CHEN. Face micro-expression recognition based on ResNet with object mask[J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(1): 303-313.

图/表 18

图1

图2

图3

图4

图5

表1

3D ResNet网络参数"

网络层	参数	输出通道数	输出尺寸
Convolution	4 $×$ 4 $×$ 4 ,S₁=4,S₂=2	32	16 $×$ 56 $×$ 56
Res block (1)	$1 × 1 × 1, S 1 = 1, S 2 = 1 3 × 3 × 3, S 1 = 1, S 2 = 2 1 × 1 × 1, S 1 = 1, S 2 = 1 1 × 1 × 1, S 1 = 1, S 2 = 1 × 2$	64	16 $×$ 56 $×$ 56
Max?pool1	2 $×$ 2 $×$ 2, S₁=2, S₂=2	64	8 $×$ 28 $×$ 28
Res block (2)	$1 × 1 × 1, S 1 = 1, S 2 = 1 3 × 3 × 3, S 1 = 1, S 2 = 2 1 × 1 × 1, S 1 = 1, S 2 = 1 1 × 1 × 1, S 1 = 1, S 2 = 1 × 2$	128	8 $×$ 28 $×$ 28
Max?pool2	2 $×$ 2 $×$ 2, S₁=2, S₂=2	128	4 $×$ 14 $×$ 14
Res block (3)	$1 × 1 × 1, S 1 = 1, S 2 = 1 3 × 3 × 3, S 1 = 1, S 2 = 2 1 × 1 × 1, S 1 = 1, S 2 = 1 1 × 1 × 1, S 1 = 1, S 2 = 1 × 2$	256	4 $×$ 14 $×$ 14
Max?pool3	2 $×$ 2 $×$ 2, S₁=2, S₂=2	256	2 $×$ 7 $×$ 7
Res block (4)	$1 × 1 × 1, S 1 = 1, S 2 = 1 3 × 3 × 3, S 1 = 1, S 2 = 2 1 × 1 × 1, S 1 = 1, S 2 = 1 1 × 1 × 1, S 1 = 1, S 2 = 1 × 2$	512	2 $×$ 7 $×$ 7
Avg?pool4	4 $×$ 4 $×$ 4, S₁=2, S₂=2	512	1 $×$ 4 $×$ 4
Flatten	-	-	8192
Fully connected layer	-	-	512
Softmax	-	-	-

表1

图6

表2

图7

图8

图9

图10

图11

图12

图13

图14

图15

表3

参考文献 28

1	Wu Q, Shen X, Fu X. Micro-expression and its applications[J]. Advances in Psychological Science, 2010, 18(9): 1359-1368.
2	Pfister T, Li X, Zhao G, et al. Recognising spontaneous facial micro-expressions[C]∥International Conference on Computer Vision, Barcelona, Spain, 2011: 1449-1456.
3	Shen X B, Wu Q, Fu X L. Effects of the duration of expressions on the recognition of micro-expressions[J]. Journal of Zhejiang University Science B:Biomedicine & Biotechnology, 2012, 13(3): 221-230.
4	Yan W J, Wu Q, Liang J, et al. How fast are the leaked facial expressions: the duration of micro-expressions[J]. Journal of Nonverbal Behavior, 2013, 37(4): 217-230.
5	Russell T A, Chu E, Phillips M L. A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool[J]. British Journal of Clinical Psychology, 2006, 45(4): 579-583.
6	贲晛烨, 杨明强, 张鹏, 等. 微表情自动识别综述[J]. 计算机辅助设计与图形学学报, 2014, 26(9): 1385-1395.
	Xian-ye Ben, Yang Ming-qiang, Zhang Peng, et al. Survey on automatic micro expression recognition methods[J]. Journal of Computer-Aided Design & Computer Graphics, 2014, 26(9): 1385-1395.
7	Zhao G, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 915-928.
8	Wang Y, See J, Phan W, et al. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition[C]∥Asian Conference on Computer Vision, Singapore, 2015: 525-537.
9	Huang X H, Wang S J, Zhao G Y, et al. Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection[C]∥IEEE International Conference on Computer Vision Workshops(ICCV), Santiago, Chile, 2015: 1-9.
10	Li X, Hong X, Moilanen A, et al. Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods[J]. IEEE Transactions on Affective Computing, 2018, 9(4): 563-577.
11	Ben X, Zhang P, Yan R, et al. Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation[J]. Neural Computing and Applications, 2016, 27(8): 2629-2646.
12	Liu Y J, Zhang J K, Yan W J, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4): 299-310.
13	Happy S L, Routray A. Fuzzy histogram of optical flow orientations for micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2017, 10(3): 394-406.
14	Kim D H, Baddar W J, Ro Y M. Micro-Expression recognition with expression-state constrained spatio-temporal feature representations[C]∥Proceedings of the ACM on Multimedia Conference, Amsterdam, The Netherlands, 2016: 382-386.
15	Li J, Wang Y, See J, et al. Micro-expression recognition based on 3D flow convolutional neural network[J]. Pattern Analysis and Applications, 2019, 22(4): 1331-1339.
16	Khor H Q, See J, Liong S T, et al. Dual-stream shallow networks for facial Micro-expression recognition[C]∥ IEEE International Conference on Image Processing, Taipei,China, 2019: 36-40.
17	Zhou Z, Zhao G, Guo Y, et al. An image-based visual speech animation system[J]. IEEE Trans on Circuits and Systems for Video Technology, 2012, 22(10): 1420-1432.
18	Wu H Y, Rubinstein M, Shih E, et al. Eulerian video magnification for revealing subtle changes in the world[J]. ACM Transactions on Graphics, 2012, 31(4): 1-8.
19	Cohn J F, Ambadar Z, Ekman P. Observer-based measurement of facial expression with the facial action coding system[J]. Neuroscience Letters, 2007, 394(3): 203-221.
20	Kazemi V, Josephine S. One millisecond face alignment with an ensemble of regression trees[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014: 1867-1874.
21	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]∥IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, NV, USA, 2016: 770-778.
22	Li Y, Huang X, Zhao G. Can micro-expression be recognized based on single apex frame?[C]∥IEEE International Conference on Image Processing(ICIP), Athens, Greece, 2018: 3094-3098.
23	Yan W J, Li X, Wang S J, et al. CASME II: an improved spontaneous micro-expression database and the baseline evaluation[J]. PLoS One, 2014, 9(1): e86041.
24	Li X, Pfister T, Huang X, et al. A spontaneous micro-expression database: Inducement, collection and baseline[C]∥IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China, 2013: 1-6.
25	Pfister T, Li X, Zhao G, et al. Recognising spontaneous facial micro-expressions[C]∥International Computer on Computer Vision, Barcelona, Spain, 2011: 1449-1456.
26	Huang X, Zhao G, Hong X, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[J]. Neurocomputing, 2016, 175: 564-578.
27	Liong S T, See J, Wong K, et al. Less is more: Micro-expression recognition from video using apex frame[J]. Signal Processing: Image Communication, 2018, 62: 82-92.
28	Ben X, Jia X, Yan R, et al. Learning effective binary descriptors for micro-expression recognition transferred by macroinformation[J]. Pattern Recognition Letters, 2018, 107: 50-58.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

方法	CASME II		SMIC
方法	准确率/%	F1?score	准确率/%	F1?score
LBP?TOP	40.9	0.369	45.7	0.461
STCLQP	58.4	-	-	-
LBP?SIP	46.6	0.448	44.5	0.449
STLBP?IP	59.5	-	-	-
Bi?WOOF	57.9	0.611	62.2	0.620
3D?FCNN	59	-	55.5	-
CNN+LSTM	61	-	-	-
DSSN	70.8	0.730	63.4	0.646
本文方法	77.3	0.774	72.4	0.721

[1]	王小玉,胡鑫豪,韩昌林. 基于生成对抗网络的人脸铅笔画算法[J]. 吉林大学学报(工学版), 2021, 51(1): 285-292.
[2]	车翔玖,董有政. 基于多尺度信息融合的图像识别改进算法[J]. 吉林大学学报(工学版), 2020, 50(5): 1747-1754.
[3]	周炳海,何朝旭. 基于线边集成超市的混流装配线动态物料配送调度[J]. 吉林大学学报(工学版), 2020, 50(5): 1809-1817.
[4]	蒋磊,管仁初. 基于多目标进化算法的人才质量模糊综合评价系统设计[J]. 吉林大学学报(工学版), 2020, 50(5): 1856-1861.
[5]	李阳,李硕,井丽巍. 基于贝叶斯模型与机器学习算法的金融风险网络评估模型[J]. 吉林大学学报(工学版), 2020, 50(5): 1862-1869.
[6]	赵宏伟,刘晓涵,张媛,范丽丽,龙曼丽,臧雪柏. 基于关键点注意力和通道注意力的服装分类算法[J]. 吉林大学学报(工学版), 2020, 50(5): 1765-1770.
[7]	管乃彦,郭娟利. 基于姿态估计算法的组件感知自适应模型[J]. 吉林大学学报(工学版), 2020, 50(5): 1850-1855.
[8]	刘洲洲,尹文晓,张倩昀,彭寒. 基于离散优化算法和机器学习的传感云入侵检测[J]. 吉林大学学报(工学版), 2020, 50(2): 692-702.
[9]	王晓辉,吴禄慎,陈华伟. 基于法向量距离分类的散乱点云数据去噪[J]. 吉林大学学报(工学版), 2020, 50(1): 278-288.
[10]	张笑东,夏筱筠,吕海峰,公绪超,廉梦佳. 大数据网络并行计算环境中生理数据流动态负载均衡[J]. 吉林大学学报(工学版), 2020, 50(1): 247-254.
[11]	金顺福,郄修尘,武海星,霍占强. 基于新型休眠模式的云虚拟机分簇调度策略及性能优化[J]. 吉林大学学报(工学版), 2020, 50(1): 237-246.
[12]	陈蔓,钟勇,李振东. 隐低秩结合低秩表示的多聚焦图像融合[J]. 吉林大学学报(工学版), 2020, 50(1): 297-305.
[13]	邓钧忆,刘衍珩,冯时,赵荣村,王健. 基于GSPN的Ad⁃hoc网络性能和安全平衡[J]. 吉林大学学报(工学版), 2020, 50(1): 255-261.
[14]	王铁君,王维兰. 基于本体的唐卡图像标注方法[J]. 吉林大学学报(工学版), 2020, 50(1): 289-296.
[15]	李雄飞,王婧,张小利,范铁虎. 基于SVM和窗口梯度的多焦距图像融合方法[J]. 吉林大学学报(工学版), 2020, 50(1): 227-236.