Multimodal temporal machine learning for bipolar disorder and depression recognition

Ceccarelli, F. and Mahmoud, M. (2022) Multimodal temporal machine learning for bipolar disorder and depression recognition. Pattern Analysis and Applications, 25(3), pp. 493-504. (doi: 10.1007/s10044-021-01001-y)

Abstract

Mental disorder is a serious public health concern that affects the life of millions of people throughout the world. Early diagnosis is essential to ensure timely treatment and to improve the well-being of those affected by a mental disorder. In this paper, we present a novel multimodal framework to perform mental disorder recognition from videos. The proposed approach employs a combination of audio, video and textual modalities. Using recurrent neural network architectures, we incorporate the temporal information in the learning process and model the dynamic evolution of the features extracted for each patient. For multimodal fusion, we propose an efficient late fusion strategy based on a simple feed-forward neural network that we call adaptive nonlinear judge classifier. We evaluate the proposed framework on two mental disorder datasets. On both, the experimental results demonstrate that the proposed framework outperforms the state-of-the-art approaches. We also study the importance of each modality for mental disorder recognition and infer interesting conclusions about the temporal nature of each modality. Our findings demonstrate that careful consideration of the temporal evolution of each modality is of crucial importance to accurately perform mental disorder recognition.

• Item Type: Articles
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Mahmoud, Dr Marwa
• Authors: Ceccarelli, F., and Mahmoud, M.
• College/School: College of Science and Engineering > School of Computing Science
• Journal Name: Pattern Analysis and Applications
• Publisher: Springer
• ISSN: 1433-7541
• ISSN (Online): 1433-755X
• Published Online: 18 June 2021
• Copyright Holders: Copyright © 2021 The Authors
• First Published: First published in Pattern Analysis and Applications 25(3): 493-504
• Publisher Policy: Reproduced in accordance with the publisher copyright policy