Is intent-aware expected reciprocal rank sufficient to evaluate diversity?

Leelanupab, T., Zuccon, G. and Jose, J. M. (2013) Is intent-aware expected reciprocal rank sufficient to evaluate diversity? In: Serdyukov, P. (ed.) Advances in Information Retrieval. Series: Lecture notes in computer science (7814). Springer, pp. 738-742. ISBN 9783642369728 (doi: 10.1007/978-3-642-36973-5_73)

Full text not currently available from Enlighten.

Publisher's URL: http://dx.doi.org/10.1007/978-3-642-36973-5_73

Abstract

In this paper we define two models of users that require diversity in search results; these models are theoretically grounded in the notion of intrinsic and extrinsic diversity. We then examine Intent-Aware Expected Reciprocal Rank (ERR-IA), one of the official measures used to assess diversity in TREC 2011-12, with respect to the proposed user models. By analyzing ranking preferences as expressed by the user models and those estimated by ERR-IA, we investigate whether ERR-IA assesses document rankings according to the requirements of the diversity retrieval task expressed by the two models. Empirical results demonstrate that ERR-IA neglects query-intents coverage by attributing excessive importance to redundant relevant documents. ERR-IA behavior is contrary to the user models that require measures to first assess diversity through the coverage of intents, and then assess the redundancy of relevant intents. Furthermore, diversity should be considered separately from document relevance and the documents positions in the ranking.

Item Type:Book Sections
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Jose, Professor Joemon and Leelanupab, Mr Teerapong and Zuccon, Mr Guido
Authors: Leelanupab, T., Zuccon, G., and Jose, J. M.
College/School:College of Science and Engineering > School of Computing Science
Journal Name:Lecture Notes in Computer Science
Publisher:Springer
ISSN:0302-9743
ISSN (Online):1611-3349
ISBN:9783642369728

University Staff: Request a correction | Enlighten Editors: Update this record