Task-based evaluation of context-sensitive referring expressions in human-robot dialogue

Foster, M. E. , Giuliani, M. and Isard, A. (2014) Task-based evaluation of context-sensitive referring expressions in human-robot dialogue. Language, Cognition and Neuroscience, 29(8), pp. 1018-1034. (doi: 10.1080/01690965.2013.855802)

Abstract

The standard referring-expression generation task involves creating stand-alone descriptions intended solely to distinguish a target object from its context. However, when an artificial system refers to objects in the course of interactive, embodied dialogue with a human partner, this is a very different setting; the references found in situated dialogue are able to take into account the aspects of the physical, interactive and task-level context, and are therefore unlike those found in corpora of stand-alone references. Also, the dominant method of evaluating generated references involves measuring corpus similarity. In an interactive context, though, other extrinsic measures such as task success and user preference are more relevant – and numerous studies have repeatedly found little or no correlation between such extrinsic metrics and the predictions of commonly used corpus-similarity metrics. To explore these issues, we introduce a humanoid robot designed to cooperate with a human partner on a joint construction task. We then describe the context-sensitive reference-generation algorithm that was implemented for use on this robot, which was inspired by the referring phenomena found in the Joint Construction Task corpus of human–human joint construction dialogues. The context-sensitive algorithm was evaluated through two user studies comparing it to a baseline algorithm, using a combination of objective performance measures and subjective user satisfaction scores. In both studies, the objective task performance and dialogue quality were found to be the same for both versions of the system; however, in both cases, the context-sensitive system scored more highly on subjective measures of interaction quality.

• Item Type: Articles
• Additional Information: The research leading to these results has received funding fromthe European Union’s Sixth Framework Programme (FP6/2003–2006) under grant agreements no. 003747, JAST: Joint ActionScience and Technology and no. 045388, INDIGO: Interactionwith Personality and Dialogue Enabled Robots, and from theSeventh Framework Programme (FP7/2007–2013) under grantagreement no. 270435, JAMES: Joint Action for MultimodalEmbodied Social Systems.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Foster, Dr Mary Ellen
• Authors: Foster, M. E., Giuliani, M., and Isard, A.
• College/School: College of Science and Engineering > School of Computing Science
• Journal Name: Language, Cognition and Neuroscience
• Publisher: Taylor & Francis
• ISSN: 2327-3798
• ISSN (Online): 2327-3801
• Published Online: 31 October 2013
• Copyright Holders: Copyright © 2013 Taylor and Francis
• First Published: First published in Language, Cognition and Neuroscience 29(8): 1018-1034
• Publisher Policy: Reproduced in accordance with the publisher copyright policy