Untangling Result List Refinement and Ranking Quality: a Framework for Evaluation and Prediction

He, J., Bron, M., de Vries, A., Azzopardi, L. and de Rijke, M. (2015) Untangling Result List Refinement and Ranking Quality: a Framework for Evaluation and Prediction. In: SIGIR 2015: 38th Annual ACM SIGIR Conference, Santiago, Chile, 09-13 Aug 2015, pp. 293-302. ISBN 978145033621 (doi: 10.1145/2766462.2767740)

Abstract

Traditional batch evaluation metrics assume that user interaction with search results is limited to scanning down a ranked list. However, modern search interfaces come with additional elements supporting result list refinement (RLR) through facets and filters, making user search behavior increasingly dynamic. We develop an evaluation framework that takes a step beyond the interaction assumption of traditional evaluation metrics and allows for batch evaluation of systems with and without RLR elements. In our framework we model user interaction as switching between different sublists. This provides a measure of user effort based on the joint effect of user interaction with RLR elements and result quality. We validate our framework by conducting a user study and comparing model predictions with real user performance. Our model predictions show significant positive correlation with real user effort. Further, in contrast to traditional evaluation metrics, the predictions using our framework, of when users stand to benefit from RLR elements, reflect findings from our user study. Finally, we use the framework to investigate under what conditions systems with and without RLR elements are likely to be effective. We simulate varying conditions concerning ranking quality,users, task and interface properties demonstrating a cost-effectiveway to study whole system performance.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Azzopardi, Dr Leif
• Authors: He, J., Bron, M., de Vries, A., Azzopardi, L., and de Rijke, M.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 978145033621
• Copyright Holders: Copyright © 2015 Association for Computing Machinery
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher.