Agents, Simulated Users and Humans

Maxwell, D. and Azzopardi, L. (2016) Agents, Simulated Users and Humans. In: 25th ACM International on Conference on Information and Knowledge Management, Indianapolis, IN, USA, 24-28 Oct 2016, pp. 731-740. ISBN 9781450340731 (doi: 10.1145/2983323.2983805)

Abstract

Most of the current models that are used to simulate users in Interactive Information Retrieval (IIR) lack realism and agency. Such models generally make decisions in a stochastic manner, without recourse to the actual information encountered or the underlying information need. In this paper, we develop a more sophisticated model of the user that includes their cognitive state within the simulation. The cognitive state maintains data about what the simulated user knows, has done and has seen, along with representations of what it considers attractive and relevant. Decisions to inspect or judge are then made based upon the simulated user's current state, rather than stochastically. In the context of ad-hoc topic retrieval, we evaluate the quality of the simulated users and agents by comparing their behaviour and performance against 48 human subjects under the same conditions, topics, time constraints, costs and search engine. Our findings show that while naive configurations of simulated users and agents substantially outperform our human subjects, their search behaviour is notably different from actual searchers. However, more sophisticated search agents can be tuned to act more like actual searchers providing greater realism. This innovation advances the state of the art in simulation, from simulated users towards autonomous agents. It provides a much needed step forward enabling the creation of more realistic simulations, while also motivating the development of more advanced cognitive agents and tools to help support and augment human searchers. Future work will focus not only on the pragmatics of tuning and training such agents for topic retrieval, but will also look at developing agents for other tasks and contexts such as collaborative search and slow search.

• Item Type: Conference Proceedings
• Additional Information: The lead author is funded by the UK Government through the EPSRC, grant number 1367507.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Azzopardi, Dr Leif and Maxwell, Mr David
• Authors: Maxwell, D., and Azzopardi, L.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781450340731
• Copyright Holders: Copyright © 2016 The Authors
• First Published: First published in Proceedings of the 25th ACM International on Conference on Information and Knowledge Management 731-740
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher