Journal of Jilin University(Engineering and Technology Edition) ›› 2022, Vol. 52 ›› Issue (4): 897-909.doi: 10.13229/j.cnki.jdxbgxb20200950

Previous Articles    

Spontaneous micro-expression recognition based on STA-LSTM

Da-xiang LI1,2(),Meng-si CHEN1,Ying LIU1,2   

  1. 1.College of Communication and Information Engineering，Xi'an University of Posts and Telecommunications，Xi'an 710121，China
    2.Ministry of Public Security Key Laboratory of Electronic Information Application Technology for Scene Investigation，Xi'an University of Posts and Telecommunications，Xi'an 710121，China
  • Received:2020-12-09 Online:2022-04-01 Published:2022-04-20

RICH HTML

 12   

PDF (PC)

225

Abstract:

In order to address the problem that most people are psychologically in sub-healthy state， a bidirectional Long Short-Term Memory （LSTM） network based on spatio-temporal attention is designed to realize the feature extraction and micro-expression recognition for finding out the emotion that people try to conceal. The network consists of three main parts： the bidirectional LSTM module， the spatial attention module and the temporal attention module. Using the micro-expression video as input， the learning network can selectively focus on action units that are significantly different for each frame， and pay different levels of attention to different frames. A new regularized cross-entropy loss function is also designed to further optimize the network， taking into account the correlation between modules. Finally， comparative experiments were performed on CASME， CASMEⅡ， CAS（ME）2， and SAMM. The experimental results show that the proposed method can improve the accuracy of Micro-Expression recognition and is superior to other methods.

Key words: computer application, micro-expression recognition, long short-term memory, spatial attention, temporal attention

CLC Number: 

  • TP391

Fig.1

STA-LSTM network"

Fig.2

LSTM neuronal structure"

Fig.3

Bidirectional LSTM structure"

Table 1

Emotional classification of CASME、CASMEⅡ、CAS（ME）2 and SAMM"

类别CASMECASMEⅡCAS（ME）2SAMM
总数175249341149
高兴83215124
愤怒--10120
厌恶4363898
恐惧---7
悲伤-8--
惊讶1920-13
压抑3827--
紧张67---
蔑视----
其他-99-77

Fig.4

Illustration of a video frames in happy micro-expression"

Fig.5

Loss function curve"

Fig.6

Influence on accuracy of weight attenuation coefficient λ1 and λ2"

Fig.7

Accuracy performance evaluation of different network modules"

Table 2

Experimental results on CASME，CASMEⅡ and CAS（ME）2"

方 法CASMECASMEⅡCAS（ME）2
F1ACCF1ACCF1ACC
LBP?TOP48.7048.8952.4856.6846.6947.72
LBP?SIP46.3446.1150.2653.8545.5045.56
LOCP?TOP51.1050.0054.5656.6847.8948.88
HOOF49.6849.7040.1042.8046.2045.50
FDM41.1242.0240.0841.9640.3242.26
CNN?LSTM64.3262.1658.9660.9859.9860.55
VGG?1135.6035.4240.2943.5743.7844.29
VGG?1635.7936.5940.6744.2944.0244.29
VGG?1936.2636.5940.9844.2944.0044.28
ResNet75.1976.3971.2974.4972.2874.48
STA?LSTM84.9787.6084.0486.3383.5983.67

Table 3

Experimental results on SAMM"

方 法F1ACC
LBP?TOP40.0341.28
Bi?WOOF52.2451.29
CapsuleNet62.3760.02
OFF?ApexNet54.3454.33
Dual?Inception59.4257.18
STSTNet66.0868.24
Mirco?Attention40.3034.00
ATNet49.6048.20
STA?LSTM78.4982.67

Fig.8

Confusion matrix of standard spontaneous micro-expression databases"

1 Haggard E A, Isaacs K S. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy[M]. Boston: Springer, 1966: 154-165.
2 Ekman P, Friesen W V. Nonverbal leakage and clues to deception[J]. Psychiatry-interpersonal & Biological Processes, 1969, 32(1): 88-106.
3 Peng M, Wu Z, Zhang Z, et al. From macro to micro expression recognition: deep learning on small datasets using transfer learning[C]∥The 13th IEEE International Conference on Automatic Face & Gesture Recognition, Xi'an,China,2018: 657-661.
4 Yao L, Xiao X, Cao R, et al. Three stream 3d CNN with SE block for micro-expression recognition[C]∥ International Conference on Computer Engineering and Application, Guangzhou, China, 2020: 439-443.
5 贲晛烨, 杨明强, 张鹏, 等. 微表情自动识别综述[J].计算机辅助设计与图形学学报, 2014, 26(9): 1385-1395.
Xian-ye Ben, Yang Ming-qiang, Zhang Peng, et al. Survey on automatic micro expression recognition methods[J]. Journal of Computer-Aided Design & Computer Graphics, 2014, 26(9): 1385-1395.
6 Yan W J, Wu Q, Liu Y J, et al. CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces[C]∥The 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, Shanghai, China, 2013: 1-7.
7 Yan W J, Li X, Wang S J, et al. CASME II: an improved spontaneous micro-expression database and the baseline evaluation[J]. Plos One, 2014, 9(1): No. e86041.
8 Qu F B, Wang S J, Yan W J, et al. CAS(ME)2: a database for spontaneous macro-expression and micro-expression spotting and recognition[J]. IEEE Transactions on Affective Computing, 2018, 9(4): 424-436.
9 Davison A K, Lansley C, Costen N, et al. SAMM: a spontaneous micro-facial movement dataset[J]. IEEE Transactions on Affective Computing, 2018, 9(1): 116-129.
10 Li X, Pfister T, Huang X, et al. A spontaneous micro-expression database: inducement, collection and baseline[C]∥The 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, Shanghai, China, 2013: 1-6.
11 Pfister T, Li X B, Zhao G, et al. Recognising spontaneous facial micro-expressions[C]∥International Conference on Computer Vision, Barcelona, Spain, 2011: 1449-1456.
12 Zhao G Y, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 915-928.
13 Huang X, Zhao G, Hong X, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[J]. Neurocomputing, 2016, 175(A): 564-578.
14 Wang Y, See J, Phan W, et al. LBP with six intersection points: reducing redundant information in lbp-top for micro-expression recognition[C]∥The 12th Asian Conference on Computer Vision, Singapore, 2014: 525-537.
15 Ben X, Zhang P, Yan R, et al. Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation[J]. Neural Computing and Applications, 2016, 27(8): 2629-2646.
16 Xu F, Zhang J, Wang J Z. Microexpression identification and categorization using a facial dynamics map[J]. IEEE Transactions on Affective Computing, 2017: 254-267.
17 Liu Y J, Zhang J K, Yan W J, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4): 299-310.
18 Fu J L, Zheng H L, Mei T. Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition[C]∥IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA, 2017: 4476-4484.
19 Lin T Y, Roychowdhury A, Maji S. Bilinear CNN models for fine-grained visual recognition[C]∥IEEE International Conference on Computer Vision, Santiago, Chile, 2015: 1449-1457.
20 Byeon Y H, Kwak K C. Facial expression recognition using 3d convolutional neural network [J]. International Journal of Advanced Computer Science and Applications, 2014, 5(12): 107-112.
21 Kim D H, Baddar W J, Ro Y M. Micro-Expression recognition with expression-stateconstrained spatio-temporal feature representations[C]∥Proceedings of the 24th ACM International Conference on Multimedia. New York, United States, 2016: 382-386.
22 Peng M, Wang C, Chen T, et al. Dual temporal scale convolutional neural network for micro-expression recognition[J]. Frontiers in Psychology, 2017: 1745-1757.
23 Patel D, Hong X P, Zhao G Y. Selective deep features for micro-expression recognition[C]∥The 23rd International Conference on Pattern Recognition, Cancun, Mexico, 2017: 2258-2263.
24 Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate[C]∥ International Conference on Learning Representations, San Diego USA, 2015.
25 Xiao T, Xu Y, Yang K, et al. The application of two-level attention models in deep convolutional neural network for fine-grained image classification[C]∥IEEE Conference on Computer Vision and Pattern, Recognition, Boston, USA, 2015: 842-850.
26 Wang F, Jiang M, Qian C, et al. Residual attention network for image classification[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 2017: 6450-6458.
27 Sharma S, Kiros R, Salakhutdinov R. Action recognition using visual attention[C]∥Neural Information Processing Systems (NIPS) Time Series Workshop, London,UK,2017: 1-11.
28 Stollenga M, Masci J, Gomez F, et al. Deep networks with internal selective attention through feedback connections[C]∥Proceedings of the 27th International Conference on Neural Information Processing Systems, Manno-Lugano, Switzerland, 2014: 3545-3553.
29 Paul Ekman, Friesen Wallace V. Facial Action Coding System:a Technique for the Measurement of Facial Movement[M]. Palo Alto: Consulting Psychologists Press, 1978.
30 Deng W H, Hu J N, Guo J. Compressive binary patterns: designing a robust binary face descriptor with random-field eigenfilters[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(3): 758-767.
31 King D E. Dlib-ml: a machine learning toolkit[J]. Machine Learning Research, 2009, 10: 1755-1758.
32 Zhou Z H, Zhao G Y, Guo Y M, et al. An image-based visual speech animation system[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(10): 1420-1432.
33 Li X, Pfister T, Huang X, et al. A Spontaneous micro-expression database: inducement, collection and baseline[C]∥The 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, Shanghai, China, 2013.
34 See J, Yap M H, Li J, et al. MEGC 2019 – the second facial micro-expressions grand challenge[C]∥The 14th IEEE International Conference on Automatic Face & Gesture Recognition, Lille, France, 2019.
35 Wang Y, See J, Phan W, et al. LBP with Six intersection points: reducing redundant information in LBP-TOP for micro-expression recognition[C]∥ACCV, Singapore,2014: 525-537.
36 Chan C H, Goswami B, Kittler J, et al. Local ordinal contrast pattern histograms for spatiotemporal, lip-based speaker authentication[C]∥The 4th IEEE International Conference on Biometrics: Theory, Applications and Systems, Washington, DC, USA, 2012: 602-612.
37 Chaudhry R, Ravichandran A, Hager G, et al. Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions[C]∥IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, USA, 2010: 1932-1939.
38 Xu F, Zhang J P, Wang J Z. Microexpression identification and categorization using a facial dynamics map[J]. IEEE Transactions on Affective Computing, 2017, 8(2): 254-267.
39 Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[C]∥Computer Vision and Pattern Recognition,Singapore,2015: 1-14.
40 He K H, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]∥IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770-778.
41 Zhao G Y, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2007, 29(6): 915-928.
42 Liong S T, See J, Wong K S, et al. Less is more: micro-expression recognition from video using apex frame[J]. Signal Processing: Image Communication, 2018, 62: 82-92.
43 Zhou L, Mao Q, Xue L. Dual-inception network for cross-database micro-expression recognition[C]∥The 14th IEEE International Conference on Automatic Face & Gesture Recognition, Lille, France, 2019.
44 Peng M, Wang C, Bi T, et al. A novel apex-time network for cross-dataset micro-expression recognition[C]∥The 8th International Conference on Affective Computing and Intelligent Interaction, Cambridge, UK, 2019.
45 Quang N V, Chun J, Tokuyama T. CapsuleNet for Micro-Expression Recognition[C]∥The 14th IEEE International Conference on Automatic Face & Gesture Recognition, Lille, France, 2019.
46 Liong S T, Gan Y S, See J, et al. Shallow triple stream three-dimensional CNN (STSTNET) for micro-expression recognition[C]∥The 14th IEEE International Conference on Automatic Face & Gesture Recognition, Lille, France, 2019.
47 Wang C Y, Peng M, Bi T, et al. Micro-attention for micro-expression recognition[J]. Neurocomputing, 2020, 410: 354-362.
[1] Ming LIU,Yu-hang YANG,Song-lin ZOU,Zhi-cheng XIAO,Yong-gang ZHANG. Application of enhanced edge detection image algorithm in multi-book recognition [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(4): 891-896.
[2] Xiao-hui WEI,Yan-wei MIAO,Xing-wang WANG. Rhombus sketch： adaptive and more accurate sketch for streaming data [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(4): 874-884.
[3] Shi-min FANG. Multiple source data selective integration algorithm based on frequent pattern tree [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(4): 885-890.
[4] Xue-zhi WANG,Qing-liang LI,Wen-hui LI. Spatio⁃temporal model of soil moisture prediction integrated with transfer learning [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 675-683.
[5] Su-ming KANG,Ye-e ZHANG. Hadoop⁃based local timing link prediction algorithm across social networks [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 626-632.
[6] Xue WANG,Zhan-shan LI,Ying-da LYU. Medical image segmentation based on multi⁃scale context⁃aware and semantic adaptor [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 640-647.
[7] Ji-hong OUYANG,Ze-qi GUO,Si-guang LIU. Dual⁃branch hybrid attention decision net for diabetic retinopathy classification [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 648-656.
[8] Lin MAO,Feng-zhi REN,Da-wei YANG,Ru-bo ZHANG. Two⁃way feature pyramid network for panoptic segmentation [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(3): 657-665.
[9] You QU,Wen-hui LI. Single-stage rotated object detection network based on anchor transformation [J]. Journal of Jilin University(Engineering and Technology Edition), 2022, 52(1): 162-173.
[10] Hong-wei ZHAO,Dong-sheng HUO,Jie WANG,Xiao-ning LI. Image classification of insect pests based on saliency detection [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2174-2181.
[11] Zhou-zhou LIU,Qian-yun ZHANG,Xin-hua MA,Han PENG. Compressed sensing signal reconstruction based on optimized discrete differential evolution algorithm [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2246-2252.
[12] Sheng-sheng WANG,Jing-yu CHEN,Yi-nan LU. COVID⁃19 chest CT image segmentation based on federated learning and blockchain [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2164-2173.
[13] Dong-ming SUN,Liang HU,Yong-heng XING,Feng WANG. Text fusion based internet of things service recommendation for trigger⁃action programming pattern [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2182-2189.
[14] Jun-cong LIN,Jun LEI,Meng CHEN,Shi-hui GUO,Xing GAO,Ming-hong LIAO. Real⁃time camera planning for dynamic multiple targets considering cinematographic visual properties [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2154-2163.
[15] Li-li REN,Zhi-jun WANG,Dong-mei YAN. Improved multi⁃verse algorithm with combined slime mould foraging behavior [J]. Journal of Jilin University(Engineering and Technology Edition), 2021, 51(6): 2190-2197.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright © Journal of Jilin University(Engineering and Technology Edition), All Rights Reserved.
Tel: +86-431-85095297　Fax: +86-431-85094074　E-mail: xbgxb@jlu.edu.cn
Powered by Beijing Magtech Co. Ltd