A 'Pointwise-Query, Listwise-Document' based Query Performance Prediction Approach

Datta, S., MacAvaney, S. , Ganguly, D. and Greene, D. (2022) A 'Pointwise-Query, Listwise-Document' based Query Performance Prediction Approach. In: SIGIR 2022: 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, 11-15 Jul 2022, pp. 2148-2153. ISBN 9781450387323 (doi: 10.1145/3477495.3531821)

[img] Text
268513.pdf - Accepted Version

997kB

Abstract

The task of Query Performance Prediction (QPP) in Information Retrieval (IR) involves predicting the relative effectiveness of a search system for a given input query. Supervised approaches for QPP, such as NeuralQPP are often trained on pairs of queries to capture their relative retrieval performance. However, pointwise approaches, such as the recently proposed BERT-QPP, are generally preferable for efficiency reasons. In this paper, we propose a novel end-to-end neural cross-encoder-based approach that is trained pointwise on individual queries, but listwise over the top ranked documents (split into chunks). In contrast to prior work, the network is then trained to predict the number of relevant documents in each chunk for a given query. Our method is thus a split-n-merge technique that instead of predicting the likely number of relevant documents in the top-k, rather predicts the number of relevant documents for each fixed chunk size p(p<k) and then aggregates them for QPP on top-k. Experiments demonstrate that our method is significantly more effective than other supervised and unsupervised QPP approaches yielding improvements of up to 30% on the TREC-DL'20 dataset and by nearly 9% for the MS MARCO Dev set.

Item Type:Conference Proceedings
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Ganguly, Dr Debasis and MacAvaney, Dr Sean
Authors: Datta, S., MacAvaney, S., Ganguly, D., and Greene, D.
College/School:College of Science and Engineering > School of Computing Science
ISBN:9781450387323
Copyright Holders:Copyright © 2022 Association for Computing Machinery
First Published:First published in SIGIR 2022: 45th International ACM SIGIR Conference on Research and Development in Information Retrieval: 2148-2153
Publisher Policy:Reproduced in accordance with the publisher copyright policy
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record