Comparisons of Expression Phases Using Local Binary Pattern Histograms for Microexpression Recognition
Abstract
Microexpression is an emotional representation occurring spontaneously and cannot be controlled consciously. It is temporary (short duration) with subtle movements, making it difficult to detect with the naked eye. Microexpressions’ muscle movements are generated in only a few small areas of the face, so observation of specific areas results in faster computation time and provides important information compared to observation of the entire face. This research proposes reducing the observation area and phase for microexpression recognition. The observed areas in the Chinese Academy of Science Micro-Expressions (CASME II) dataset are left and right eyebrows, right and left eyes, and mouth. The observation phase of microexpressions included analyzing the comparison in the onset to offset phase (“fullOAO”) and in the onset, apex, and offset phase (“OAO”). Feature extraction was performed using a simple local binary patterns histogram (LBPH) method, which can represent local features in the facial area. The best result of the proposed method was the “fullOAO” phase with an accuracy of 96.8% (using support vector machine-radial basis function, SVM-RBF) and an average computation time of 0.192 ms per frame and 10.473 ms per video. In “OAO” phase type, an accuracy of 87.7% was achieved with a computation time of 0.159 ms per frame and 0.576 ms per video. The difference in accuracy and computation time between the two-phase types occurs because the number of frames in “fullOAO” type is greater than in “OAO”, resulting in a different amount of processing time and feature extraction data. However, the 9% decrease in accuracy does not significantly affect the accuracy since the accuracy rate is still relatively good, above 80%. Furthermore, the correct measurement for computation time was the time taken to process each frame in the input video. Therefore, the proposed method can produce fast computation time and relatively accurate recognition.
References
L. Zhou, X. Shao, and Q. Mao, “A Survey of Micro-Expression Recognition,” Image, Vis. Comput., Vol. 105, pp. 1–11, Jan. 2021, doi: 10.1016/j.imavis.2020.104043.
P. Ekman, “Lie Catching and Microexpressions,” in The Philosophy of Deception, C. Martin, Ed., New York, USA: Oxford University Press, 2009.
P. Ekman and W.V. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto, USA: Consulting Psychologists Press, 1978.
M. Peng, Z. Wu, Z. Zhang, and T. Chen, “From Macro to Micro Expression Recognition: Deep Learning on Small Datasets Using Transfer Learning,” 2018 13th IEEE Int. Conf. Autom. Face Gesture Recognit. (FG 2018), 2018, pp. 657–661, doi: 10.1109/FG.2018.00103.
C.H. Yap, C. Kendrick, and M.H. Yap, “SAMM Long Videos: A Spontaneous Facial Micro- and Macro-Expressions Dataset,” 2019, arXiv:1911.01519.
P. Ekman and W.V. Friesen, “Nonverbal Leakage and Clues to Deception,” Psychiatry, Vol. 32, No. 1, pp. 88–106, 1969, doi: 10.1080/00332747.1969.11023575.
H.-X. Xie, L. Lo, H.-H. Shuai, and W.-H. Cheng, “AU-assisted Graph Attention Convolutional Network for Micro-Expression Recognition,” Proc. 28th ACM Int. Conf. Multimed., 2020, pp. 2871–2880, doi: 10.1145/3394171.3414012.
J. Ma, H. Tang, W.-L. Zheng, and B.-L. Lu, “Emotion Recognition Using Multimodal Residual LSTM Network,” Proc. 27th ACM Int. Conf. Multimed., 2019, pp. 176–183, doi: 10.1145/3343031.3350871.
I.P. Adegun and H.B. Vadapalli, “Facial Micro-Expression Recognition: A Machine Learning Approach,” Sci. Afr., Vol. 8, pp. 1–14, Jul. 2020, doi: 10.1016/j.sciaf.2020.e00465.
Y. Zhu, Z. Chen, and F. Wu, “Multimodal Deep Denoise Framework for Affective Video Content Analysis,” Proc. 27th ACM Int. Conf. Multimed., 2019, pp. 130–138.
A.M. Buhari et al., “FACS-Based Graph Features for Real-Time Micro-Expression Recognition,” J. Imaging, Vol. 6, No. 12, pp. 1–20, Dec. 2020, doi: 10.3390/jimaging6120130.
S.-J. Wang et al., “Micro-Expression Recognition Using Robust Principal Component Analysis and Local Spatiotemporal Directional Features,” in Computer Vision - ECCV 2014 Workshops, L. Agapito, M.M. Bronstein, and C. Rother, Eds., Cham, Switzerland: Springer International Publishing, 2015, pp. 325–338, doi: 10.1007/978-3-319-16178-5_23.
Y. Wang, J. See, R. C.-W. Phan, and Y.-H. Oh, “LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition,” in Computer Vision – ACCV 2014, D. Cremers, I. Reid, H. Saito, and M.-H. Yang, Eds., Cham, Switzerland: Springer International Publishing, 2015, pp. 525–537, doi: 10.1007/978-3-319-16865-4_34.
P. Zhang et al., “Micro-Expression Recognition System,” Optik, Vol. 127, No. 3, pp. 1395–1400, Feb. 2016, doi: 10.1016/j.ijleo.2015.10.217.
M.N. Patil, B. Iyer, and R. Arya, “Performance Evaluation of PCA and ICA Algorithm for Facial Expression Recognition Application,” in Proceedings of Fifth International Conference on Soft Computing for Problem Solving, SocProS 2015, Vol. 1, M. Pant et al., Eds., Singapore, Singapore: Springer, 2016, pp. 965–976, doi: 10.1007/978-981-10-0448-3_81.
W.-L. Chao, J.-J. Ding, and J.-Z. Liu, “Facial Expression Recognition Based on Improved Local Binary Pattern and Class-Regularized Locality Preserving Projection,” Signal Process., Vol. 117, pp. 1–10, Dec. 2015, doi: 10.1016/j.sigpro.2015.04.007.
X.R. Soh, V.M. Baskaran, A.M. Buhari, and R.C.-W. Phan, “A Real Time Micro-Expression Detection System with LBP-TOP on a Many-Core Processor,” 2017 Asia-Pacific Signal, Inf. Process. Assoc. Annu. Summit, Conf. (APSIPA ASC), 2017, pp. 309–315, doi: 10.1109/APSIPA.2017.8282041.
X. Hong, Y. Xu, and G. Zhao, “LBP-TOP: A Tensor Unfolding Revisit,” in Computer Vision – ACCV 2016 Workshops, C.-S. Chen, J. Lu, and K.-K. Ma, Eds., Cham, Switzerland: Springer International Publishing, 2017, pp. 513–527, doi: 10.1007/978-3-319-54407-6_34.
N. Samadiani et al., “A Review on Automatic Facial Expression Recognition Systems Assisted by Multimodal Sensor Data,” Sens., Vol. 19, No. 8, pp. 1–27, Apr. 2019, doi: 10.3390/s19081863.
P. Choirina, U.D. Rosiani, and I.M. Fitriani, “Pengenalan Ekspresi Mikro Wajah Berdasarkan Point Feature Tracking Menggunakan Fase Apex pada Database Ekspresi Mikro,” Edu Komputika J., Vol. 9, No. 1, pp. 28–36, Jun. 2022, doi: 10.15294/edukomputika.v9i1.56600.
U.D. Rosiani et al., “A Novel Approach on Motion Estimation for Micro-Expression Recognition Using Phase Only Correlation with All Block Search (POC-ABS),” Int. J. Intell. Eng., Syst., Vol. 13, No. 6, pp. 546–559, 2020, doi: 10.22266/ijies2020.1231.48.
U.D. Rosiani, P. Choirina, S. Sumpeno, and M. Hery P., “Menuju Pengenalan Ekspresi Mikro: Pendeteksian Komponen Wajah Menggunakan Discriminative Response Map Fitting,” J. Nas. Tek. Elekt., Teknol. Inf., Vol. 7, No. 2, pp. 204–211, May 2018, doi: 10.22146/jnteti.v7i2.424.
S.P.T. Reddy, S.T. Karri, S.R. Dubey, and S. Mukherjee, “Spontaneous Facial Micro-Expression Recognition Using 3D Spatiotemporal Convolutional Neural Networks,” 2019 Int. Joint Conf. Neural Netw. (IJCNN), 2019, pp. 1–8, doi: 10.1109/IJCNN.2019.8852419.
W.-J. Yan et al., “CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation,” PLoS ONE, Vol. 9, No. 1, pp. 1–8, Jan. 2014, doi: 10.1371/journal.pone.0086041.
“CASME II Database,” [Online], http://casme.psych.ac.cn/casme/e2, access date: 26-Jun-2023.
M. Xu, D. Chen, and G. Zhou, “Real-Time Face Recognition Based on Dlib,” in Innovative Computing, C.-T. Yang, Y. Pei, and J.-W. Chang, Eds., Singapore, Singapore: Springer, 2020, pp. 1451–1459, doi: 10.1007/978-981-15-5959-4_177.
T.Q. Vinh and N.T.N. Anh, “Real-Time Face Mask Detector Using YOLOv3 Algorithm and Haar Cascade Classifier,” 2020 Int. Conf. Adv. Comput., Appl. (ACOMP), 2020, pp. 146–149, doi: 10.1109/ACOMP50827.2020.00029.
N.M. Parsania, K.H. Solanki, and A.R. Mehta. (3-4 Sep. 2021). Innovative Approach for Fingerprint Recognition Using LBP and PCA Algorithms. Presented at 1st Int. Conf. Adv. Inf. Technol., Commun. (IC-AITC), [Online], https://www.youtube.com/watch?v=8x3vsNyfLg0
V. Esmaeili, M.M. Feghhi, and S.O. Shahdi, “Micro-Expression Recognition Using Histogram of Image Gradient Orientation on Diagonal Planes,” 2021 5th Int. Conf. Pattern Recognit., Image Anal. (IPRIA), 2021, pp. 1–5, doi: 10.1109/IPRIA53572.2021.9483500.
D. Rahmawati et al., “The Design of Facial Expression Detection System to Determine the Level of Customer Satisfaction Using K-Nearest Neighbor Method,” MATEC Web Conf., Vol. 372, 2022, pp. 1–5, doi: 10.1051/matecconf/202237206002.
H. Pan et al., “Hierarchical Support Vector Machine for Facial Micro-Expression Recognition,” Multimed. Tools, Appl., Vol. 79, pp. 31451–31465, Nov. 2020, doi: 10.1007/s11042-020-09475-4.
A.K. Davison, W. Merghani, and M.H. Yap, “Objective Classes for Micro-Facial Expression Recognition,” J. Imag., Vol. 4, No. 10, pp. 1–13, Oct. 2018, doi: 10.3390/jimaging4100119.
S.-T. Liong, J. See, K. Wong, and R C.-W. Phan, “Less is More: Micro-Expression Recognition from Video Using Apex Frame,” Signal Process. Image Commun., Vol. 62, pp. 82–92, Mar. 2018, doi: 10.1016/j.image.2017.11.006.
Y.S. Gan et al., “OFF-ApexNet on Micro-Expression Recognition System,” Signal Process. Image Commun., Vol. 74, pp. 129–139, May 2019, doi: 10.1016/j.image.2019.02.005.
© Jurnal Nasional Teknik Elektro dan Teknologi Informasi, under the terms of the Creative Commons Attribution-ShareAlike 4.0 International License.
1.png)
