CMCF: An Architecture for Realtime Gesture Generation by Clustering Gestures by Motion and Communicative Function

Saund, C., Birladeanu, A. and Marsella, S. (2021) CMCF: An Architecture for Realtime Gesture Generation by Clustering Gestures by Motion and Communicative Function. In: 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS '21), 3-7 May 2021, pp. 1136-1144. ISBN 9781450383073

[img] Text
253967.pdf - Accepted Version

5MB

Publisher's URL: https://dl.acm.org/doi/10.5555/3463952.3464084

Abstract

Gestures augment speech by performing a variety of communicative functions in humans and virtual agents, and are often related to speech by complex semantic, rhetorical, prosodic, and affective elements. In this paper we briefly present an architecture for human-like gesturing in virtual agents that is designed to realize complex speech-to-gesture mappings by exploiting existing machine-learning based parsing tools and techniques to extract these functional elements from speech. We then deeply explore the rhetorical branch of this architecture, objectively assessing specifically whether existing rhetorical parsing techniques can classify gestures into classes with distinct movement properties. To do this, we take a corpus of spontaneously generated gestures and correlate their movement to co-speech utterances. We cluster gestures based on their rhetorical properties, and then by their movement. Our objective analysis suggests that some rhetorical structures are identifiable by our movement features while others require further exploration. We explore possibilities behind these findings and propose future experiments that may further reveal nuances of the richness of the mapping between speech and motion. This work builds towards a real-time gesture generator which performs gestures that effectively convey rich communicative functions.

Item Type:Conference Proceedings
Additional Information:The work in this article has been supported by United Kingdom Research and Innovation through the UKRI Centre for Doctoral Training in Socially Intelligent Artificial Agents (EP/S02266X/1) and by the Royal Wolfson Society Award (WRM/FT/170004).
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Birladeanu, Andrei and Marsella, Professor Stacy
Authors: Saund, C., Birladeanu, A., and Marsella, S.
College/School:College of Medical Veterinary and Life Sciences > School of Psychology & Neuroscience
College of Science and Engineering
ISBN:9781450383073
Copyright Holders:Copyright © 2021 International Foundation for Autonomous Agents and Multiagent Systems
First Published:First published in Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS '21), 1136–1144
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
303764EPSRC CDT - Socially Intelligent Artificial AgentsAlessandro VinciarelliEngineering and Physical Sciences Research Council (EPSRC)EP/S02266X/1Computing Science