An Analysis of Variations in the Effectiveness of Query Performance Prediction

Ganguly, D. , Datta, S., Mitra, M. and Greene, D. (2022) An Analysis of Variations in the Effectiveness of Query Performance Prediction. In: 44th European Conference on Information Retrieval (ECIR 2022), Stavanger, Norway, 10-14 Apr 2022, pp. 215-229. ISBN 9783030997359 (doi: 10.1007/978-3-030-99736-6_15)

Abstract

A query performance predictor estimates the retrieval effectiveness of a system for a given query. Query performance prediction (QPP) algorithms are themselves evaluated by measuring the correlation between the predicted effectiveness and the actual effectiveness of a system for a set of queries. This generally accepted framework for judging the usefulness of a QPP method includes a number of sources of variability. For example, “actual effectiveness” can be measured using different metrics, for different rank cut-offs. The objective of this study is to identify some of these sources, and investigate how variations in the framework can affect the outcomes of QPP experiments. We consider this issue not only in terms of the absolute values of the evaluation metrics being reported (e.g., Pearson’s r, Kendall’s τ), but also with respect to the changes in the ranks of different QPP systems when ordered by the QPP metric scores. Our experiments reveal that the observed QPP outcomes can vary considerably, both in terms of the absolute evaluation metric values and also in terms of the relative system ranks. We report the combinations of QPP evaluation metric and experimental settings that are likely to lead to smaller variations in the observed results.

• Item Type: Conference Proceedings
• Keywords: Query performance prediction, variations in QPP results, QPP reproducibility.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Ganguly, Dr Debasis
• Authors: Ganguly, D., Datta, S., Mitra, M., and Greene, D.
• Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9783030997359
• Published Online: 05 April 2022
• Copyright Holders: Copyright © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022
• First Published: First published in Advances in Information Retrieval. ECIR 2022. Lecture Notes in Computer Science, vol 13185. Springer, Cham.
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Organisation
• Publisher