Looking at the body: automatic analysis of body gestures and self-adaptors in psychological distress

Lin, W., Orton, I., Li, Q., Pavarini, G. and Mahmoud, M. (2023) Looking at the body: automatic analysis of body gestures and self-adaptors in psychological distress. IEEE Transactions on Affective Computing, 14(2), pp. 1175-1187. (doi: 10.1109/TAFFC.2021.3101698)

Abstract

Psychological distress is a significant and growing issue in society. In particular, depression and anxiety are leading causes of disability that often go undetected or late-diagnosed. Automatic detection, assessment, and analysis of behavioural markers of psychological distress can help improve identification and support prevention and early intervention efforts. Compared to modalities such as face, head, and vocal, research investigating the use of the body modality for these tasks is relatively sparse, which is partly due to the limited available datasets and difficulty in automatically extracting useful body features. To enable our research, we have collected and analyzed a new dataset containing full body videos for interviews and self-reported distress labels. We propose a novel approach to automatically detect self-adaptors and fidgeting, a subset of self-adaptors that has been shown to correlate with psychological distress. We perform analysis on statistical body gestures and fidgeting features to explore how distress levels affect behaviors. We then propose a multi-modal approach that combines different feature representations using Multi-modal Deep Denoising Auto-Encoders and Improved Fisher Vector Encoding. We demonstrate that our proposed model, combining audio-visual features with detected fidgeting behavioral cues, can successfully predict depression and anxiety in the dataset.

• Item Type: Articles
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Mahmoud, Dr Marwa
• Authors: Lin, W., Orton, I., Li, Q., Pavarini, G., and Mahmoud, M.
• College/School: College of Science and Engineering > School of Computing Science
• Journal Name: IEEE Transactions on Affective Computing
• Publisher: IEEE
• ISSN: 1949-3045
• ISSN (Online): 1949-3045
• Published Online: 04 August 2021
• Copyright Holders: Copyright © 2021 IEEE
• First Published: First published in IEEE Transactions on Affective Computing 14(2): 1175-1187
• Publisher Policy: Reproduced in accordance with the publisher copyright policy