Multimodal temporal machine learning for bipolar disorder and depression recognition

Ceccarelli, F. and Mahmoud, M. (2022) Multimodal temporal machine learning for bipolar disorder and depression recognition. Pattern Analysis and Applications, 25(3), pp. 493-504. (doi: 10.1007/s10044-021-01001-y)

[img] Text
259854.pdf - Accepted Version



Mental disorder is a serious public health concern that affects the life of millions of people throughout the world. Early diagnosis is essential to ensure timely treatment and to improve the well-being of those affected by a mental disorder. In this paper, we present a novel multimodal framework to perform mental disorder recognition from videos. The proposed approach employs a combination of audio, video and textual modalities. Using recurrent neural network architectures, we incorporate the temporal information in the learning process and model the dynamic evolution of the features extracted for each patient. For multimodal fusion, we propose an efficient late fusion strategy based on a simple feed-forward neural network that we call adaptive nonlinear judge classifier. We evaluate the proposed framework on two mental disorder datasets. On both, the experimental results demonstrate that the proposed framework outperforms the state-of-the-art approaches. We also study the importance of each modality for mental disorder recognition and infer interesting conclusions about the temporal nature of each modality. Our findings demonstrate that careful consideration of the temporal evolution of each modality is of crucial importance to accurately perform mental disorder recognition.

Item Type:Articles
Glasgow Author(s) Enlighten ID:Mahmoud, Dr Marwa
Authors: Ceccarelli, F., and Mahmoud, M.
College/School:College of Science and Engineering > School of Computing Science
Journal Name:Pattern Analysis and Applications
ISSN (Online):1433-755X
Published Online:18 June 2021
Copyright Holders:Copyright © 2021 The Authors
First Published:First published in Pattern Analysis and Applications 25(3): 493-504
Publisher Policy:Reproduced in accordance with the publisher copyright policy

University Staff: Request a correction | Enlighten Editors: Update this record