COMEX: A Multi-task Benchmark for Knowledge-grounded COnversational Media EXploration

Tun, Z. Y., Speggiorin, A., Dalton, J. and Stamper, M. (2022) COMEX: A Multi-task Benchmark for Knowledge-grounded COnversational Media EXploration. In: Conversational User Interfaces (CUI 2022), Glasgow, UK, 26-28 July 2022, p. 11. ISBN 9781450397391 (doi: 10.1145/3543829.3543830)

Abstract

Open-domain conversational interaction with news, podcasts, and other types of heterogeneous content remains an open challenge. Interactive agents must support information access in a way that is fair, impartial, and true to the content and knowledge discussed. To facilitate this, systems building on interactive retrieval from knowledge-grounded media are a controllable and known base for experimentation. A conversational media agent should retrieve relevant content, understand key concepts in the content through grounding to a knowledge base, and enable exploration by offering to discuss a topic further or progress to describe related topics. In this work, we release a new multi-task benchmark on COnversational Media EXploration (COMEX) to measure knowledge-grounded conversational content exploration. It consists of a heterogeneous semantically annotated media corpus and topic-specific data for 1) entity Wikification and salience, 2) conversational content ranking on heterogeneous media content, 3) background link ranking, and 4) background linking explanation. We develop COMEX with judgments and conversational interactions developed in partnership with professional editorial staff from the BBC. We study the behavior of state-of-the-art systems, with the results demonstrating significant headroom on all tasks.

• Item Type: Conference Proceedings
• Additional Information: This work is supported by a grant from The Data Lab in partnership with the BBC. Additionally, this work is supported by a Turing AI Acceleration Fellowship from the Engineering and Physical Sciences Research Council, grant number EP/V025708/1.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Speggiorin, Mr Alessandro and Dalton, Dr Jeff and Tun, Mr Zay Yar
• Authors: Tun, Z. Y., Speggiorin, A., Dalton, J., and Stamper, M.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781450397391
• Copyright Holders: Copyright © 2022 The Authors
• First Published: First published in Conversational User Interfaces (CUI 2022): 11
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Organisation
• Publisher

Project Code	Award No	Project Name	Principal Investigator	Funder's Name	Funder Ref	Lead Dept
310549		Dalton-UKRI-Turing Fellow	Jeff Dalton	Engineering and Physical Sciences Research Council (EPSRC)	EP/V025708/1	Computing Science