Dynamic emotion modeling with learnable graphs and graph inception network

Shirian, A., Tripathi, S. and Guha, T. (2022) Dynamic emotion modeling with learnable graphs and graph inception network. IEEE Transactions on Multimedia, 24, pp. 780-790. (doi: 10.1109/TMM.2021.3059169)

Abstract

Human emotion is expressed, perceived and captured using a variety of dynamic data modalities, such as speech (verbal), videos (facial expressions) and motion sensors (body gestures). We propose a generalized approach to emotion recognition that can adapt across modalities by modeling dynamic data as structured graphs. The motivation behind the graph approach is to build compact models without compromising on performance. To alleviate the problem of optimal graph construction, we cast this as a joint graph learning and classification task. To this end, we present the Learnable Graph Inception Network (L-GrIN) that jointly learns to recognize emotion and to identify the underlying graph structure in the dynamic data. Our architecture comprises multiple novel components: a new graph convolution operation, a graph inception layer, learnable adjacency, and a learnable pooling function that yields a graph-level embedding. We evaluate the proposed architecture on five benchmark emotion recognition databases spanning three different modalities (video, audio, motion capture), where each database captures one of the following emotional cues: facial expressions, speech and body gestures. We achieve state-of-the-art performance on all five databases outperforming several competitive baselines and relevant existing methods. Our graph architecture shows superior performance with significantly fewer parameters (compared to convolutional or recurrent neural networks) promising its applicability to resource-constrained devices. Our code is available at https://github.com/AmirSh15/graph_emotion_recognition.

• Item Type: Articles
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Guha, Dr Tanaya
• Authors: Shirian, A., Tripathi, S., and Guha, T.
• College/School: College of Science and Engineering > School of Computing Science
• Research Group: GIST
• Journal Name: IEEE Transactions on Multimedia
• Publisher: IEEE
• ISSN: 1520-9210
• ISSN (Online): 1941-0077
• Published Online: 15 February 2021
• Copyright Holders: Copyright © 2021 IEEE
• First Published: First published in IEEE Transactions on Multimedia 24: 780-790
• Publisher Policy: Reproduced in accordance with the publisher copyright policy