结合残差网络及目标掩膜的人脸微表情识别

doi:10.13229/j.cnki.jdxbgxb20190939

Abstract

Abstract:

Micro-expression is a kind of facial feature that can reflect the most real emotional state hidden in human heart. Existing micro-expression recognition technology often extracts a lot of feature information unrelated to micro-expression in feature extraction. In order to solve the above problem， this paper proposes a feature extraction method of specific region of face micro-expression based on deep learning. Firstly， we preprocess the original micro-expression sequence to locate the key regions of human eyes， mask the eye regions， and reduce the blinking action to extract micro-expression features. Then， Euler video magnification algorithm is used to magnify the key areas of facial micro-expression change， which makes the part of micro-expression change more obvious. Finally， the continuous micro-expression sequence is trained and recognized by 3D ResNet network. The proposed method is tested on CASME II and SMIC datasets， and he experimental results show that the accuracies are 77.3% and 72.4% respectively. Compared with the latest methods DSSN， CNN+LSTM， the accuracy is improved at least 5%.

Key words: computer application, micro-expression, 3D ResNet, object mask, Eulerian video magnification, emotion recognition

CLC Number:

TP391.4

Ming FANG,Wen-qiang CHEN. Face micro-expression recognition based on ResNet with object mask[J].Journal of Jilin University(Engineering and Technology Edition), 2021, 51(1): 303-313.

Figures/Tables 18

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Parameters of 3D ResNet network"

网络层	参数	输出通道数	输出尺寸
Convolution	4 $×$ 4 $×$ 4 ,S₁=4,S₂=2	32	16 $×$ 56 $×$ 56
Res block (1)	$1 × 1 × 1, S 1 = 1, S 2 = 1 3 × 3 × 3, S 1 = 1, S 2 = 2 1 × 1 × 1, S 1 = 1, S 2 = 1 1 × 1 × 1, S 1 = 1, S 2 = 1 × 2$	64	16 $×$ 56 $×$ 56
Max?pool1	2 $×$ 2 $×$ 2, S₁=2, S₂=2	64	8 $×$ 28 $×$ 28
Res block (2)	$1 × 1 × 1, S 1 = 1, S 2 = 1 3 × 3 × 3, S 1 = 1, S 2 = 2 1 × 1 × 1, S 1 = 1, S 2 = 1 1 × 1 × 1, S 1 = 1, S 2 = 1 × 2$	128	8 $×$ 28 $×$ 28
Max?pool2	2 $×$ 2 $×$ 2, S₁=2, S₂=2	128	4 $×$ 14 $×$ 14
Res block (3)	$1 × 1 × 1, S 1 = 1, S 2 = 1 3 × 3 × 3, S 1 = 1, S 2 = 2 1 × 1 × 1, S 1 = 1, S 2 = 1 1 × 1 × 1, S 1 = 1, S 2 = 1 × 2$	256	4 $×$ 14 $×$ 14
Max?pool3	2 $×$ 2 $×$ 2, S₁=2, S₂=2	256	2 $×$ 7 $×$ 7
Res block (4)	$1 × 1 × 1, S 1 = 1, S 2 = 1 3 × 3 × 3, S 1 = 1, S 2 = 2 1 × 1 × 1, S 1 = 1, S 2 = 1 1 × 1 × 1, S 1 = 1, S 2 = 1 × 2$	512	2 $×$ 7 $×$ 7
Avg?pool4	4 $×$ 4 $×$ 4, S₁=2, S₂=2	512	1 $×$ 4 $×$ 4
Flatten	-	-	8192
Fully connected layer	-	-	512
Softmax	-	-	-

Table 1

Fig.6

Table 2

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Fig.14

Fig.15

Table 3

References 28

1	Wu Q, Shen X, Fu X. Micro-expression and its applications[J]. Advances in Psychological Science, 2010, 18(9): 1359-1368.
2	Pfister T, Li X, Zhao G, et al. Recognising spontaneous facial micro-expressions[C]∥International Conference on Computer Vision, Barcelona, Spain, 2011: 1449-1456.
3	Shen X B, Wu Q, Fu X L. Effects of the duration of expressions on the recognition of micro-expressions[J]. Journal of Zhejiang University Science B:Biomedicine & Biotechnology, 2012, 13(3): 221-230.
4	Yan W J, Wu Q, Liang J, et al. How fast are the leaked facial expressions: the duration of micro-expressions[J]. Journal of Nonverbal Behavior, 2013, 37(4): 217-230.
5	Russell T A, Chu E, Phillips M L. A pilot study to investigate the effectiveness of emotion recognition remediation in schizophrenia using the micro-expression training tool[J]. British Journal of Clinical Psychology, 2006, 45(4): 579-583.
6	贲晛烨, 杨明强, 张鹏, 等. 微表情自动识别综述[J]. 计算机辅助设计与图形学学报, 2014, 26(9): 1385-1395.
	Xian-ye Ben, Yang Ming-qiang, Zhang Peng, et al. Survey on automatic micro expression recognition methods[J]. Journal of Computer-Aided Design & Computer Graphics, 2014, 26(9): 1385-1395.
7	Zhao G, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 915-928.
8	Wang Y, See J, Phan W, et al. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition[C]∥Asian Conference on Computer Vision, Singapore, 2015: 525-537.
9	Huang X H, Wang S J, Zhao G Y, et al. Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection[C]∥IEEE International Conference on Computer Vision Workshops(ICCV), Santiago, Chile, 2015: 1-9.
10	Li X, Hong X, Moilanen A, et al. Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods[J]. IEEE Transactions on Affective Computing, 2018, 9(4): 563-577.
11	Ben X, Zhang P, Yan R, et al. Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation[J]. Neural Computing and Applications, 2016, 27(8): 2629-2646.
12	Liu Y J, Zhang J K, Yan W J, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4): 299-310.
13	Happy S L, Routray A. Fuzzy histogram of optical flow orientations for micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2017, 10(3): 394-406.
14	Kim D H, Baddar W J, Ro Y M. Micro-Expression recognition with expression-state constrained spatio-temporal feature representations[C]∥Proceedings of the ACM on Multimedia Conference, Amsterdam, The Netherlands, 2016: 382-386.
15	Li J, Wang Y, See J, et al. Micro-expression recognition based on 3D flow convolutional neural network[J]. Pattern Analysis and Applications, 2019, 22(4): 1331-1339.
16	Khor H Q, See J, Liong S T, et al. Dual-stream shallow networks for facial Micro-expression recognition[C]∥ IEEE International Conference on Image Processing, Taipei,China, 2019: 36-40.
17	Zhou Z, Zhao G, Guo Y, et al. An image-based visual speech animation system[J]. IEEE Trans on Circuits and Systems for Video Technology, 2012, 22(10): 1420-1432.
18	Wu H Y, Rubinstein M, Shih E, et al. Eulerian video magnification for revealing subtle changes in the world[J]. ACM Transactions on Graphics, 2012, 31(4): 1-8.
19	Cohn J F, Ambadar Z, Ekman P. Observer-based measurement of facial expression with the facial action coding system[J]. Neuroscience Letters, 2007, 394(3): 203-221.
20	Kazemi V, Josephine S. One millisecond face alignment with an ensemble of regression trees[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014: 1867-1874.
21	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]∥IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Las Vegas, NV, USA, 2016: 770-778.
22	Li Y, Huang X, Zhao G. Can micro-expression be recognized based on single apex frame?[C]∥IEEE International Conference on Image Processing(ICIP), Athens, Greece, 2018: 3094-3098.
23	Yan W J, Li X, Wang S J, et al. CASME II: an improved spontaneous micro-expression database and the baseline evaluation[J]. PLoS One, 2014, 9(1): e86041.
24	Li X, Pfister T, Huang X, et al. A spontaneous micro-expression database: Inducement, collection and baseline[C]∥IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China, 2013: 1-6.
25	Pfister T, Li X, Zhao G, et al. Recognising spontaneous facial micro-expressions[C]∥International Computer on Computer Vision, Barcelona, Spain, 2011: 1449-1456.
26	Huang X, Zhao G, Hong X, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[J]. Neurocomputing, 2016, 175: 564-578.
27	Liong S T, See J, Wong K, et al. Less is more: Micro-expression recognition from video using apex frame[J]. Signal Processing: Image Communication, 2018, 62: 82-92.
28	Ben X, Jia X, Yan R, et al. Learning effective binary descriptors for micro-expression recognition transferred by macroinformation[J]. Pattern Recognition Letters, 2018, 107: 50-58.

Related Articles 15

[1]	Xiao-yu WANG,Xin-hao HU,Chang-lin HAN. Face pencil drawing algorithms based on generative adversarial network [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(1): 285-292.
[2]	Xiang-jiu CHE,You-zheng DONG. Improved image recognition algorithm based on multi⁃scale information fusion [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(5): 1747-1754.
[3]	Bing-hai ZHOU,Zhao-xu HE. Dynamic material handling scheduling for mixed⁃model assembly lines based on line⁃integrated supermarkets [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(5): 1809-1817.
[4]	Lei JIANG,Ren-chu GUAN. Design of fuzzy comprehensive evaluation system for talent quality based on multi⁃objective evolutionary algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(5): 1856-1861.
[5]	Yang LI,Shuo LI,Li-wei JING. Estimate model based on Bayesian model and machine learning algorithms applicated in financial risk assessment [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(5): 1862-1869.
[6]	Hong-wei ZHAO,Xiao-han LIU,Yuan ZHANG,Li-li FAN,Man-li LONG,Xue-bai ZANG. Clothing classification algorithm based on landmark attention and channel attention [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(5): 1765-1770.
[7]	Nai-yan GUAN,Juan-li GUO. Component awareness adaptive model based on attitude estimation algorithms [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(5): 1850-1855.
[8]	Zhou-zhou LIU,Wen-xiao YIN,Qian-yun ZHANG,Han PENG. Sensor cloud intrusion detection based on discrete optimization algorithm and machine learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(2): 692-702.
[9]	Xiao-hui WANG,Lu-shen WU,Hua-wei CHEN. Denoising of scattered point cloud data based on normal vector distance classification [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(1): 278-288.
[10]	Xiao-dong ZHANG,Xiao-jun XIA,Hai-feng LYU,Xu-chao GONG,Meng-jia LIAN. Dynamic load balancing of physiological data flow in big data network parallel computing environment [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(1): 247-254.
[11]	Shun-fu JIN,Xiu-chen QIE,Hai-xing WU,Zhan-qiang HUO. Clustered virtual machine allocation strategy in cloud computing based on new type of sleep-mode and performance optimization [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(1): 237-246.
[12]	Man CHEN,Yong ZHONG,Zhen-dong LI. Multi-focus image fusion based on latent low⁃rank representation combining low⁃rank representation [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(1): 297-305.
[13]	Jun-yi DENG,Yan-heng LIU,Shi FENG,Rong-cun ZHAO,Jian WANG. GSPN⁃based model to evaluate the performance and securi tytradeoff in Ad-hoc network [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(1): 255-261.
[14]	Tie-jun WANG,Wei-lan WANG. Thangka image annotation based on ontology [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(1): 289-296.
[15]	Xiong-fei LI,Jing WANG,Xiao-li ZHANG,Tie-hu FAN. Multi-focus image fusion based on support vector machines and window gradient [J]. Journal of Jilin University(Engineering and Technology Edition), 2020, 50(1): 227-236.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 0

No Suggested Reading articles found!

方法	CASME II		SMIC
方法	准确率/%	F1?score	准确率/%	F1?score
LBP?TOP	40.9	0.369	45.7	0.461
STCLQP	58.4	-	-	-
LBP?SIP	46.6	0.448	44.5	0.449
STLBP?IP	59.5	-	-	-
Bi?WOOF	57.9	0.611	62.2	0.620
3D?FCNN	59	-	55.5	-
CNN+LSTM	61	-	-	-
DSSN	70.8	0.730	63.4	0.646
本文方法	77.3	0.774	72.4	0.721

Face micro-expression recognition based on ResNet with object mask

RICH HTML

PDF (PC)