Real-time Inference of Mental States From Facial Expressions and Upper Body Gestures

Baltrušaitis, T., McDuff, D., Banda, N., Mahmoud, M. , El Kaliouby, R., Robinson, P. and Picard, R. (2011) Real-time Inference of Mental States From Facial Expressions and Upper Body Gestures. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), Santa Barbara, CA, USA, 21-25 Mar 2011, pp. 909-914. ISBN 9781424491414 (doi: 10.1109/FG.2011.5771372)

Abstract

We present a real-time system for detecting facial action units and inferring emotional states from head and shoulder gestures and facial expressions. The dynamic system uses three levels of inference on progressively longer time scales. Firstly, facial action units and head orientation are identified from 22 feature points and Gabor filters. Secondly, Hidden Markov Models are used to classify sequences of actions into head and shoulder gestures. Finally, a multi level Dynamic Bayesian Network is used to model the unfolding emotional state based on probabilities of different gestures. The most probable state over a given video clip is chosen as the label for that clip. The average F1 score for 12 action units (AUs 1, 2, 4, 6, 7, 10, 12, 15, 17, 18, 25, 26), labelled on a frame by frame basis, was 0.461. The average classification rate for five emotional states (anger, fear, joy, relief, sadness) was 0.440. Sadness had the greatest rate, 0.64, anger the smallest, 0.11.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Mahmoud, Dr Marwa
• Authors: Baltrušaitis, T., McDuff, D., Banda, N., Mahmoud, M., El Kaliouby, R., Robinson, P., and Picard, R.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781424491414
• Published Online: 19 May 2011