Unsupervised Query Performance Prediction for Neural Models with Pairwise Rank Preferences

Singh, A., Ganguly, D. , Datta, S. and McDonald, C. (2023) Unsupervised Query Performance Prediction for Neural Models with Pairwise Rank Preferences. In: 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR23), Taipei, Taiwan, 23-27 July 2023, pp. 2486-2490. ISBN 9781450394086 (doi: 10.1145/3539618.3592082)

Abstract

A query performance prediction (QPP) method predicts the effectiveness of an IR system for a given query. While unsupervised approaches have been shown to work well for statistical IR models, it is likely that these approaches would yield limited effectiveness for neural ranking models (NRMs) because the retrieval scores of these models lie within a short range unlike their statistical counterparts. In this work, we propose to leverage a pairwise inference-based NRM's (specifically, DuoT5) output to accumulate evidences on the pairwise believes of one document ranked above the other. We hypothesize that the more consistent these pairwise likelihoods are, the higher is the likelihood of the retrieval to be of better quality, thus yielding a higher QPP score. We conduct our experiments on the TREC-DL dataset leveraging pairwise likelihoods from an auxiliary model DuoT5. Our experiments demonstrate that the proposed method called Pairwise Rank Preference-based QPP (QPP-PRP) leads to significantly better results than a number of standard unsupervised QPP baselines on several NRMs.

• Item Type: Conference Proceedings
• Additional Information: The third author was partially supported by Science Foundation Ireland (SFI) grant number SFI/12/RC/2289_P2.
• Keywords: Unsupervised query performance prediction, neural ranking models, pairwise rank preferences.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Ganguly, Dr Debasis
• Authors: Singh, A., Ganguly, D., Datta, S., and McDonald, C.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781450394086