Building culturally-valid dynamic facial expressions for a conversational virtual agent using human perception

Chen, C. , Garrod, O. G. B., Ince, R. A. A. , Foster, M. E. , Schyns, P. G. and Jack, R. E. (2020) Building culturally-valid dynamic facial expressions for a conversational virtual agent using human perception. In: Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, Scotland, Virtual Event, UK, 20 - 22 October 2020, ISBN 9781450375863 (doi: 10.1145/3383652.3423913)

Full text not currently available from Enlighten.

Abstract

Facial expressions are used to facilitate daily interactions including conversations, in every culture. However, little is known about which specific facial expressions convey conversational messages, which limits the social signalling capabilities of conversational virtual agents. We address this critical knowledge gap by modeling the facial expressions of key conversational messages directly from cultural perception, and transferring them to a conversational virtual agent. Using a novel data-driven psychology-based method, we modelled facial expressions of 'thinking,' 'interested,' 'bored' and 'confused' directly from the cultural perception of 40 participants across two distinct cultures (Western European, East Asian). We then transferred these facial expression models to a popular conversational virtual agent and validated them with a new group of cultural participants. Results showed that, in both cultures, the majority of our culturally derived conversational facial expression models successfully transferred to the agent. A further cross-cultural analysis of the conversational facial expressions showed systematic similarities (e.g., eye brow raising in 'interested') and differences (e.g., Westerners use horizontal mouth stretch and East Asians use vertical mouth opening to show confusion) that could facilitate or hinder cross-cultural communication. Our results demonstrate the power of using a culturally sensitive perception-based psychological approach to develop psychologically impactful facial expressions for conversational virtual agents. We anticipate that our facial expression models will enhance virtual agent social signalling capabilities and their global marketability.

Item Type:Conference Proceedings
Additional Information:Article No.: 14
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Jack, Professor Rachael and Garrod, Dr Oliver and Schyns, Professor Philippe and Chen, Dr Chaona and Foster, Dr Mary Ellen and Ince, Dr Robin
Authors: Chen, C., Garrod, O. G. B., Ince, R. A. A., Foster, M. E., Schyns, P. G., and Jack, R. E.
College/School:College of Medical Veterinary and Life Sciences > School of Psychology & Neuroscience
College of Science and Engineering > School of Psychology
Journal Name:Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents
Publisher:ACM
ISBN:9781450375863
Published Online:19 October 2020
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record