Leelanupab, T., Zuccon, G. and Jose, J. M. (2013) Is intent-aware expected reciprocal rank sufficient to evaluate diversity? In: Serdyukov, P. (ed.) Advances in Information Retrieval. Series: Lecture notes in computer science (7814). Springer, pp. 738-742. ISBN 9783642369728 (doi: 10.1007/978-3-642-36973-5_73)
Full text not currently available from Enlighten.
Publisher's URL: http://dx.doi.org/10.1007/978-3-642-36973-5_73
Abstract
In this paper we define two models of users that require diversity in search results; these models are theoretically grounded in the notion of intrinsic and extrinsic diversity. We then examine Intent-Aware Expected Reciprocal Rank (ERR-IA), one of the official measures used to assess diversity in TREC 2011-12, with respect to the proposed user models. By analyzing ranking preferences as expressed by the user models and those estimated by ERR-IA, we investigate whether ERR-IA assesses document rankings according to the requirements of the diversity retrieval task expressed by the two models. Empirical results demonstrate that ERR-IA neglects query-intents coverage by attributing excessive importance to redundant relevant documents. ERR-IA behavior is contrary to the user models that require measures to first assess diversity through the coverage of intents, and then assess the redundancy of relevant intents. Furthermore, diversity should be considered separately from document relevance and the documents positions in the ranking.
Item Type: | Book Sections |
---|---|
Status: | Published |
Refereed: | Yes |
Glasgow Author(s) Enlighten ID: | Jose, Professor Joemon and Leelanupab, Mr Teerapong and Zuccon, Mr Guido |
Authors: | Leelanupab, T., Zuccon, G., and Jose, J. M. |
College/School: | College of Science and Engineering > School of Computing Science |
Journal Name: | Lecture Notes in Computer Science |
Publisher: | Springer |
ISSN: | 0302-9743 |
ISSN (Online): | 1611-3349 |
ISBN: | 9783642369728 |
University Staff: Request a correction | Enlighten Editors: Update this record