A user behavior model for average precision and its generalization to graded judgments

Dupret, G. and Piwowarski, B. (2010) A user behavior model for average precision and its generalization to graded judgments. In: Crestania, F., Stéphane, M.-M., Chen, H.-H., Efthimiadis, E.N. and Savoy, J. (eds.) Proceedings of the 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery: New York, USA, pp. 531-538. ISBN 9781450301534 (doi: 10.1145/1835449.1835538)

Abstract

We explore a set of hypothesis on user behavior that are potentially at the origin of the (Mean) Average Precision (AP) metric. This allows us to propose a more realistic version of AP where users click non-deterministically on relevant documents and where the number of relevant documents in the collection needs not be known in advance. We then depart from the assumption that a document is either relevant or irrelevant and we use instead relevance judgment similar to editorial labels used for Discounted Cumulated Gain (DCG). We assume that clicked documents provide users with a certain level of "utility" and that a user ends a search when she gathered enough utility. Based on the query logs of a commercial search engine we show how to evaluate the utility associated with a label from the record of past user interactions with the search engine and we show how the two different user models can be evaluated based on their ability to predict accurately future clicks. Finally, based on these user models, we propose a measure that captures the relative quality of two rankings.

• Item Type: Book Sections
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Piwowarski, Dr Benjamin
• Authors: Dupret, G., and Piwowarski, B.
• Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
• College/School: College of Science and Engineering > School of Computing Science
• Publisher: Association for Computing Machinery
• ISBN: 9781450301534