On the Feasibility and Robustness of Pointwise Evaluation of Query Performance Prediction

Datta, S., Ganguly, D. , Greene, D. and Mitra, M. (2023) On the Feasibility and Robustness of Pointwise Evaluation of Query Performance Prediction. In: QPP++ 2023: Query Performance Prediction and Its Evaluation in New Tasks, Dublin, Ireland, 06 Apr 2023, pp. 7-12.

Abstract

Despite the retrieval effectiveness of queries being mutually independent of one another, the evaluation of query performance prediction (QPP) systems has been carried out by measuring rank correlation over an entire set of queries. Such a listwise approach has a number of disadvantages, notably that it does not support the common requirement of assessing QPP for individual queries. In this paper, we propose a pointwise QPP framework that allows us to evaluate the quality of a QPP system for individual queries by measuring the deviations between each prediction versus the corresponding true value, and then aggregating the results over a set of queries. Our experiments demonstrate that this new approach leads to smaller variances in QPP evaluations across a range of different target metrics and retrieval models.

• Item Type: Conference Proceedings
• Additional Information: The first and the third authors were supported by the Science Foundation Ireland (SFI) grant number SFI/12/RC/2289_P2.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Ganguly, Dr Debasis
• Authors: Datta, S., Ganguly, D., Greene, D., and Mitra, M.
• College/School: College of Science and Engineering > School of Computing Science
• ISSN: 1613-0073
• Copyright Holders: © 2023 Copyright for this paper by its authors
• First Published: First published in Proceedings of the The QPP++ 2023: Query Performance Prediction and Its Evaluation in New Tasks Workshop co-located with The 45th European Conference on Information Retrieval (ECIR)
• Publisher Policy: Reproduced under a Creative Commons license