Tackling biased baselines in the risk-sensitive evaluation of retrieval systems

Dinçer, B. T., Ounis, I. and Macdonald, C. (2014) Tackling biased baselines in the risk-sensitive evaluation of retrieval systems. Lecture Notes in Computer Science, 8416, pp. 26-38. (doi: 10.1007/978-3-319-06028-6_3)

Full text not currently available from Enlighten.

Publisher's URL: http://dx.doi.org/10.1007/978-3-319-06028-6_3


The aim of optimising information retrieval (IR) systems using a risk-sensitive evaluation methodology is to minimise the risk of performing any particular topic less effectively than a given baseline system. Baseline systems in this context determine the reference effectiveness for topics, relative to which the effectiveness of a given IR system in minimising the risk will be measured. However, the comparative risk-sensitive evaluation of a set of diverse IR systems – as attempted by the TREC 2013 Web track – is challenging, as the different systems under evaluation may be based upon a variety of different (base) retrieval models, such as learning to rank or language models. Hence, a question arises about how to properly measure the risk exhibited by each system. In this paper, we argue that no model of information retrieval alone is representative enough in this respect to be a true reference for the models available in the current state-of-the-art, and demonstrate, using the TREC 2012 Web track data, that as the baseline system changes, the resulting risk-based ranking of the systems changes significantly. Instead of using a particular system’s effectiveness as the reference effectiveness for topics, we propose several remedies including the use of mean within-topic system effectiveness as a baseline, which is shown to enable unbiased measurements of the risk-sensitive effectiveness of IR systems.

Item Type:Articles
Additional Information:Proceedings of 36th European Conference on IR Research, ECIR 2014, Amsterdam, The Netherlands, 13-16 April, 2014. ISBN: 9783319060279
Glasgow Author(s) Enlighten ID:Macdonald, Professor Craig and Ounis, Professor Iadh
Authors: Dinçer, B. T., Ounis, I., and Macdonald, C.
College/School:College of Science and Engineering > School of Computing Science
Journal Name:Lecture Notes in Computer Science
Publisher:Springer Verlag
ISSN (Online):1611-3349

University Staff: Request a correction | Enlighten Editors: Update this record