Evaluating implicit feedback models using searcher simulations

White, R.M., Ruthven, I., Jose, J.M. and Van Rijsbergen, C.J. (2005) Evaluating implicit feedback models using searcher simulations. ACM Transactions on Information Systems, 23(3), pp. 325-361. (doi: 10.1145/1080343.1080347)



Publisher's URL: http://doi.acm.org/10.1145/1080343.1080347


In this article we describe an evaluation of relevance feedback (RF) algorithms using searcher simulations. Since these algorithms select additional terms for query modification based on inferences made from searcher interaction, not on relevance information searchers explicitly provide (as in traditional RF), we refer to them as implicit feedback models. We introduce six different models that base their decisions on the interactions of searchers and use different approaches to rank query modification terms. The aim of this article is to determine which of these models should be used to assist searchers in the systems we develop. To evaluate these models we used searcher simulations that afforded us more control over the experimental conditions than experiments with human subjects and allowed complex interaction to be modeled without the need for costly human experimentation. The simulation-based evaluation methodology measures how well the models learn the distribution of terms across relevant documents (i.e., learn what information is relevant) and how well they improve search effectiveness (i.e., create effective search queries). Our findings show that an implicit feedback model based on Jeffrey's rule of conditioning outperformed other models under investigation.

Item Type:Articles
Additional Information:© ACM, 2005. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Information Systems 23(3):325-361 http://doi.acm.org/10.1145/1080343.1080347
Keywords:User simulations, evaluation, implicit feedback, relevance feedback
Glasgow Author(s) Enlighten ID:Jose, Professor Joemon and Van Rijsbergen, Professor Cornelis
Authors: White, R.M., Ruthven, I., Jose, J.M., and Van Rijsbergen, C.J.
Subjects:Q Science > QA Mathematics > QA75 Electronic computers. Computer science
College/School:College of Science and Engineering > School of Computing Science
Journal Name:ACM Transactions on Information Systems
Publisher:ACM Press
Copyright Holders:Copyright © 2005 ACM Press
First Published:First published in ACM Transactions on Information Systems 23(3):325-361
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher including additional information statement.

University Staff: Request a correction | Enlighten Editors: Update this record