Automatic analysis of naturalistic hand-over-face gestures

Mahmoud, M. , Baltrušaitis, T. and Robinson, P. (2016) Automatic analysis of naturalistic hand-over-face gestures. ACM Transactions on Intelligent Interactive Systems, 6(2), 19. (doi: 10.1145/2946796)

Full text not currently available from Enlighten.


One of the main factors that limit the accuracy of facial analysis systems is hand occlusion. As the face becomes occluded, facial features are lost, corrupted, or erroneously detected. Hand-over-face occlusions are considered not only very common but also very challenging to handle. However, there is empirical evidence that some of these hand-over-face gestures serve as cues for recognition of cognitive mental states. In this article, we present an analysis of automatic detection and classification of hand-over-face gestures. We detect hand-over-face occlusions and classify hand-over-face gesture descriptors in videos of natural expressions using multi-modal fusion of different state-of-the-art spatial and spatio-temporal features. We show experimentally that we can successfully detect face occlusions with an accuracy of 83%. We also demonstrate that we can classify gesture descriptors (hand shape, hand action, and facial region occluded) significantly better than a naïve baseline. Our detailed quantitative analysis sheds some light on the challenges of automatic classification of hand-over-face gestures in natural expressions.

Item Type:Articles
Glasgow Author(s) Enlighten ID:Mahmoud, Dr Marwa
Authors: Mahmoud, M., Baltrušaitis, T., and Robinson, P.
College/School:College of Science and Engineering > School of Computing Science
Journal Name:ACM Transactions on Intelligent Interactive Systems
ISSN (Online):2160-6463
Published Online:20 July 2016

University Staff: Request a correction | Enlighten Editors: Update this record