Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond

Owoicho, P. O., Sekulić, I., Aliannejadi, M., Dalton, J. and Crestani, F. (2023) Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond. In: 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR23), Taipei, Taiwan, 23-27 July 2023, pp. 632-642. ISBN 9781450394086 (doi: 10.1145/3539618.3591683)

Abstract

This research aims to explore various methods for assessing user feedback in mixed-initiative conversational search (CS ) systems. While CS systems enjoy profuse advancements across multiple aspects, recent research fails to successfully incorporate feedback from the users. One of the main reasons for that is the lack of system-user conversational interaction data. To this end, we propose a user simulator-based framework for multi-turn interactions with a variety of mixed-initiative CS systems. Specifically, we develop a user simulator, dubbed ConvSim, that, once initialized with an information need description, is capable of providing feedback to system's responses, as well as answering potential clarifying questions. Our experiments on a wide variety of state-of-the-art passage retrieval and neural re-ranking models show that effective utilization of user feedback can lead to 16% retrieval performance increase in terms of nDCG@3. Moreover, we observe consistent improvements as the number of feedback rounds increases (35% relative improvement in terms of nDCG@3 after three rounds). This points to a research gap in the development of specific feedback processing modules and opens a potential for significant advancements in CS. To support further research in the topic, we release over 30 000 transcripts of system-simulator interactions based on well-established CS datasets.

• Item Type: Conference Proceedings
• Additional Information: This work is supported by the Engineering and Physical Sciences Research Council (EPSRC) grant EP/V025708/1 and the 2019 Google Research Grant.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Owoicho, Paul Ogbonoko and Dalton, Dr Jeff
• Authors: Owoicho, P. O., Sekulić, I., Aliannejadi, M., Dalton, J., and Crestani, F.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781450394086
• Copyright Holders: Copyright © 2023 The Authors
• First Published: First published in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR23), pp. 632-642
• Publisher Policy: Reproduced under a Creative Commons License

Project Code	Award No	Project Name	Principal Investigator	Funder's Name	Funder Ref	Lead Dept
310549		Dalton-UKRI-Turing Fellow	Jeff Dalton	Engineering and Physical Sciences Research Council (EPSRC)	EP/V025708/1	Computing Science