Classification of facial expressions of emotional faces by people and a neural network based on an ecologically valid image database
DOI:
https://doi.org/10.21638/spbu16.2025.107Abstract
This paper investigates the problem of universal facial expression patterns for emotion valences, emotion families, and culturally specific emotions. Many studies have been conducted to address this problem, but they sometimes infer the universality of emotion expression from respondents’ recognition of emotions and use traditional image databases that do not meet the conditions of ecological validity. Four groups of theories about the origin of emotions are compared: the theory of universality of basic emotions, the theory of cultural specificity of basic emotions, the theory of social learning of emotions, and the theory of social construction of emotions. Purpose of the study: to determine whether people’s facial expressions have specific patterns for positive and negative valences of emotion (degree of positive or negative affective response), emotion families (includes basic emotion and variations of basic emotion), and specific emotions using an image database whose authors strived for ecological validity: subjects were not given a standard of facial expression patterns to demonstrate emotions. A teachable machine neural network was used to examine the expression process in a respondent-independent manner. Heterogeneous transfer learning was used. In our work, respondents and the neural network categorized images of emotional faces into 14 emotions. Respondents were more likely than the neural network to correctly classify specific emotions. Both the neural network and humans were more efficient at recognizing valence and emotion families than recognizing specific emotions. This may suggest specific patterns of emotion families and emotion valence. Weak specificity at the level of specific emotions is possible. The results are consistent with Scarantino’s “New BET” theory of basic emotions. An interpretation in line with the social learning theory of emotion is possible. The limitations of the neural network in recognizing the full range of variations in prototypical emotion expressions are discussed. Further possible cross-cultural studies to refine the results are described.
Keywords:
origin of emotion, basic emotions, constructivism, emotion classification, emotional expression
Downloads
No metrics available.
References
Литература
Барабанщиков В. А., Королькова О. А., Лободинская Е. А. Распознавание эмоций в условиях ступенчатой стробоскопической экспозиции выражений лица // Экспериментальная психология. 2018. Т. 11, № 4. С. 50–69. https://doi.org/10.17759/exppsy.2018110405
Барабанщиков В. А., Суворова Е. В. Оценка эмоционального состояния человека по его видеоизображению // Экспериментальная психология. 2020. Т. 13, № 4. С. 4–24. https://doi.org/10.17759/exppsy.2020130401
Королькова О. А., Лободинская Е. А. База видеоизображений естественных эмоциональных экспрессий: восприятие эмоций и автоматизированный анализ мимики лица // Оптический журнал. 2022. Т. 89, № 8. С. 97–103. https://doi.org/10.17759/exppsy.2021140401
Петракова А. В., Лебедева Е. И., Кузьмина Ю. В., Юрчик Е. Н. Опыт создания российской базы лиц, изображающих различные эмоции: первый этап // Психология. Журнал Высшей школы экономики. 2024. Т. 21, № 2. С. 423–431. https://doi.org/10.17323/1813 8918 2024 2 423 431
Рюмина Е. В., Карпов А. А. Аналитический обзор методов распознавания эмоций по выражениям лица человека // Научно-технический вестник информационных технологий, механики и оптики. 2020. Т. 20, № 2. С. 163–176. https://doi.org/10.17586/2226 1494 2020 20 2 163 176
Федосеева Е. В., Терехова В. А., Цесаренко О. В., Гладкова М. М. Обработка результатов токсикологических исследований в статистической программе R // Принципы экологии. 2015. № 3 (15). С. 12–26. https://doi.org/10.15393/j1.art.2015.4381
Badrulhisham N. A. S., Mangshor N. N. A. Emotion recognition using convolutional neural network (CNN) // Journal of Physics: Conference Series. IOP Publishing. 2021. Vol. 1962, no. 1. P. 012040. https://doi.org/10.1088/1742-6596/1962/1/012040
Bandura A., Walters R. H. Social learning theory. Englewood Cliffs: Prentice hall, 1977. Vol. 1. P. 141–154.
Barrett L. F. Constructing emotion // Psihologijske teme. 2011. Vol. 20, no. 3. P. 359–380.
Barrett L. F. Discrete emotions or dimensions? The role of valence focus and arousal focus // Cognition & Emotion. 1998. Vol. 12, no. 4. P. 579–599. https://doi.org/10.1080/026999398379574
Barrett L. F., Adolphs R., Marsella S., Martinez A. M., Pollak S. D. Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements // Psychological science in the public interest. 2019. Vol. 20, no. 1. P. 1–68. https://doi.org/10.1177/1529100619832930
Barrett L. F., Simmons W. K. Interoceptive predictions in the brain // Nature reviews neuroscience. 2015. Vol. 16, no. 7. P. 419–429. https://doi.org/10.1038/nrn3950
Carney M., Webster B., Alvarado I., Phillips K., Howell N., Griffith J., Jongejan J., Pitaru A., Chen A. Teachable machine: Approachable Web-based tool for exploring machine learning classification // Extended abstracts of the 2020 CHI conference on human factors in computing systems. 2020. P. 1–8. https://doi.org/10.1145/3334480.3382839
Dailey M. N., Cottrell G. W., Padgett C., Adolphs R. EMPATH: A neural network that categorizes facial expressions // Journal of cognitive neuroscience. 2002. Vol. 14, no. 8. P. 1158–1173. https://doi.org/10.1162/089892902760807177
Deng J., Dong W., Socher R., Li L. J., Li K., Fei-Fei L. ImageNet: A large-scale hierarchical image database // 2009 IEEE conference on computer vision and pattern recognition. 2009. P. 248–255. https://doi.org/10.1109/CVPR.2009.5206848
Ekman P. An argument for basic emotions // Cognition & Emotion. 1992. Vol. 6, no. 3–4. P. 169–200. https://doi.org/10.1080/02699939208411068
Ekman P. Basic emotions // Handbook of cognition and emotion. Chichester: Wiley, 1999a. P. 16.
Ekman P. Facial expressions // Handbook of cognition and emotion. Chichester: Wiley, 1999b. P. 320.
Ekman P. Universal facial expressions in emotion // Studia Psychologica. 1973. Vol. 15, no. 2. P. 140–147.
Ekman P. Universals and cultural differences in facial expressions of emotion // Nebraska symposium on motivation. Lincoln: University of Nebraska Press, 1971. P. 207–283.
Ekman P., Friesen W. V. Constants across cultures in the face and emotion // Journal of personality and social psychology. 1971. Vol. 17, no. 2. P. 124. https://doi.org/10.1037/h0030377
Gendron M., Hoemann K., Crittenden A. N., Mangola S. M., Ruark G. A., Barrett L. F. Emotion perception in Hadza hunter-gatherers // Scientific reports. 2020. Vol. 10, no. 1. P. 3867. https://doi.org/10.1038/s41598 020 60257 2
Gendron M., Roberson D., van der Vyver J. M., Barrett L. F. Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture // Emotion. 2014. Vol. 14, no. 2. P. 251. https://doi.org/10.1037/a0036052
Gruhn D., Sharifian N. Lists of emotional stimuli // Emotion measurement. Sawston: Woodhead Publishing, 2016. P. 145–164. https://doi.org/10.1016/B978 0 08 100508 8.00007-2
Kretzschmar R., Karayiannis N. B., Eggimann F. Handling class overlap with variance-controlled neural networks // Proceedings of the International Joint Conference on Neural Networks, 2003. New York: IEEE, 2003. Vol. 1. P. 517–522. https://doi.org/10.1109/IJCNN.2003.1223400
Leon E., Clarke G., Callaghan V., Sepulveda F. Real-time detection of emotional changes for inhabited environments // Computers & Graphics. 2004. Vol. 28, no. 5. P. 635–642. https://doi.org/10.1016/j.cag.2004.06.002
Oatley K., Johnson-Laird P. N. The communicative theory of emotions: Empirical tests, mental models, and implications for social interaction // Striving and feeling. Hove: Psychology Press, 2014. P. 363–393.
Russell J. A. Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies // Psychological Bulletin. 1994. Vol. 115, no. 1. P. 102. https://doi.org/10.1037/0033-2909.115.1.102
Sato W., Hyniewska S., Minemoto K., Yoshikawa S. Facial expressions of basic emotions in Japanese laypeople // Frontiers in psychology. 2019. Vol. 10. P. 420264. https://doi.org/10.3389/fpsyg.2019.00259
Scarantino A. Basic emotions, psychological construction, and the problem of variability // The psychological construction of emotion. New York: The Guilford Press, 2015. P. 334–376. https://doi.org/10.1007/s13164 020 00492 8
Scherer K. R., Wallbott H. G. Evidence for universality and cultural variation of differential emotion response patterning // Journal of personality and social psychology. 1994. Vol. 66, no. 2. P. 310. https://doi.org/10.1037/0022-3514.66.2.310
Trommsdorff G. Development of emotions as organized by culture // ISSBD Newsletter. 2006. Vol. 49, no. 1. P. 1–4.
References
Akers, R. L., Jennings, W. G. (2015). Social learning theory. The handbook of criminological theory, 230–240.
Badrulhisham, N. A. S., Mangshor, N. N. A. (2021). Emotion recognition using convolutional neural network (CNN). Journal of Physics: Conference Series, IOP Publishing, 1962, 1, 012040.
Barabanshhikov, V. A., Korol'kova, O. A., Lobodinskaja, E. A. (2018). Emotion recognition under staggered stroboscopic exposure of facial expressions. Eksperimental'naia psikhologiia, 11 (4), 50–69. (In Russian)
Barabanshhikov, V. A., Suvorova, E. V. (2020). Estimation of a person's emotional state from his video image. Eksperimental'naia psikhologiia, 13 (4), 4–24. (In Russian)
Barrett, L. F. (2011). Constructing emotion. Psihologijske teme, 20 (3), 359–380.
Barrett, L. F. (1998). Discrete emotions or dimensions? The role of valence focus and arousal focus. Cognition & Emotion, 12 (4), 579–599.
Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological science in the public interest, 20 (1), 1–68.
Barrett, L. F., Simmons, W. K. (2015). Interoceptive predictions in the brain. Nature Reviews Neuroscience, 16 (7), 419–429.
Carney, M., Webster, B., Alvarado, I., Phillips, K., Howell, N., Griffith, J., Jongejan, J., Pitaru, A., Chen, A. (2020). Teachable machine: Approachable Web-based tool for exploring machine learning classification. In: Extended abstracts of the 2020 CHI conference on human factors in computing systems (pp. 1–8).
Dailey, M. N., Cottrell, G. W., Padgett, C., Adolphs, R. (2002). EMPATH: A neural network that categorizes facial expressions. Journal of cognitive neuroscience, 14 (8), 1158–1173.
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition (pp. 248–255).
Ekman, P. (1971). Universals and cultural differences in facial expressions of emotion. In: Nebraska symposium on motivation (pp. 207–283). University of Nebraska Press.
Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6 (3–4), 169–200.
Ekman, P. (1999a). Basic emotions. In: Handbook of cognition and emotion (p. 16). Chichester: Wiley
Ekman, P. (1999b). Facial expressions. In: Handbook of cognition and emotion (p. 320). Chichester: Wiley
Ekman, P., Keltner, D. (1970). Universal facial expressions of emotion. California mental health research digest, 8 (4), 151–158.
Ekman, P., Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of personality and social psychology, 17 (2), 124.
Fedoseeva, E. V., Terekhova, V. A., Tsesarenko, O. V., Gladkova, M. M. (2015). Processing the results of toxicological studies in a statistical program R. Printsipy ekologii, 3 (15), 12–26. (In Russian)
Gendron, M., Hoemann, K., Crittenden, A. N., Mangola, S. M., Ruark, G. A., Barrett, L. F. (2020). Emotion perception in Hadza hunter-gatherers. Scientific reports, 10 (1), 3867.
Gendron, M., Roberson, D., van der Vyver, J. M., Barrett, L. F. (2014). Perceptions of emotion from facial expressions are not culturally universal: evidence from a remote culture. Emotion, 14 (2), 251.
Gruhn, D., Sharifian, N. (2016). Lists of emotional stimuli. In: Emotion measurement (pp. 145–164). Sawston, Woodhead Publishing
Korol'kova, O. A., Lobodinskaia, E. A. (2022). A video database of natural emotional expressions: emotion perception and automated analysis of facial expressions. Opticheskii zhurnal, 89 (8), 97–103. (In Russian)
Kretzschmar, R., Karayiannis, N. B., Eggimann, F. (2003). Handling class overlap with variance-controlled neural networks. In: Proceedings of the International Joint Conference on Neural Networks (vol. 1, pp. 517–522). New York: IEEE
Leon, E., Clarke, G., Callaghan, V., Sepulveda, F. (2004). Real-time detection of emotional changes for inhabited environments. Computers & Graphics, 28 (5), 635–642.
Oatley, K., Johnson-Laird, P. N. (2014). The communicative theory of emotions: Empirical tests, mental models, and implications for social interaction. In: Striving and feeling (pp. 363–393). Hove, Psychology Press
Petrakova, A. V., Lebedeva, E. I., Kuz'mina, Iu. V., Iurchik, E. N. (2024). Experience of creating a Russian database of faces depicting different emotions: The first stage. Psikhologiia. Zhurnal Vysshei shkoly ekonomiki, 21 (2), 423–431. (In Russian)
Rjumina, E. V., Karpov, A. A. (2020). Analytical review of methods for recognizing emotions from human facial expressions. Nauchno-tehnicheskii vestnik informatsionnykh tehnologii, mekhaniki i optiki, 20 (2), 163–176. (In Russian)
Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115 (1), 102.
Sato, W., Hyniewska, S., Minemoto, K., Yoshikawa, S. (2019). Facial expressions of basic emotions in Japanese laypeople. Frontiers in Psychology, 10, 259.
Scarantino, A. (2015). Basic emotions, psychological construction, and the problem of variability. In: The psychological construction of emotion (pp. 334–376). New York: The Guilford Press
Scherer, K. R., Wallbott, H. G. (1994). Evidence for universality and cultural variation of differential emotion response patterning. Journal of personality and social psychology, 66 (2), 310.
Trommsdorff, G. (2006). Development of emotions as organized by culture. ISSBD Newsletter, 49 (1), 1–4.
Downloads
Published
2025-05-21
How to Cite
Sladkoshtieva, A., Starodubtsev, A., & Petrakova, A. (2025). Classification of facial expressions of emotional faces by people and a neural network based on an ecologically valid image database. Vestnik of Saint Petersburg University. Psychology, 15(1), 116–133. https://doi.org/10.21638/spbu16.2025.107
Issue
Section
Empirical and Experimental Research
License
Articles of "Vestnik of Saint Petersburg University. Psychology" are open access distributed under the terms of the License Agreement with Saint Petersburg State University, which permits to the authors unrestricted distribution and self-archiving free of charge.
