Datta, S., MacAvaney, S. , Ganguly, D. and Greene, D. (2022) A 'Pointwise-Query, Listwise-Document' based Query Performance Prediction Approach. In: SIGIR 2022: 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, 11-15 Jul 2022, pp. 2148-2153. ISBN 9781450387323 (doi: 10.1145/3477495.3531821)
Text
268513.pdf - Accepted Version 997kB |
Abstract
The task of Query Performance Prediction (QPP) in Information Retrieval (IR) involves predicting the relative effectiveness of a search system for a given input query. Supervised approaches for QPP, such as NeuralQPP are often trained on pairs of queries to capture their relative retrieval performance. However, pointwise approaches, such as the recently proposed BERT-QPP, are generally preferable for efficiency reasons. In this paper, we propose a novel end-to-end neural cross-encoder-based approach that is trained pointwise on individual queries, but listwise over the top ranked documents (split into chunks). In contrast to prior work, the network is then trained to predict the number of relevant documents in each chunk for a given query. Our method is thus a split-n-merge technique that instead of predicting the likely number of relevant documents in the top-k, rather predicts the number of relevant documents for each fixed chunk size p(p<k) and then aggregates them for QPP on top-k. Experiments demonstrate that our method is significantly more effective than other supervised and unsupervised QPP approaches yielding improvements of up to 30% on the TREC-DL'20 dataset and by nearly 9% for the MS MARCO Dev set.
Item Type: | Conference Proceedings |
---|---|
Status: | Published |
Refereed: | Yes |
Glasgow Author(s) Enlighten ID: | Ganguly, Dr Debasis and MacAvaney, Dr Sean |
Authors: | Datta, S., MacAvaney, S., Ganguly, D., and Greene, D. |
College/School: | College of Science and Engineering > School of Computing Science |
ISBN: | 9781450387323 |
Copyright Holders: | Copyright © 2022 Association for Computing Machinery |
First Published: | First published in SIGIR 2022: 45th International ACM SIGIR Conference on Research and Development in Information Retrieval: 2148-2153 |
Publisher Policy: | Reproduced in accordance with the publisher copyright policy |
Related URLs: |
University Staff: Request a correction | Enlighten Editors: Update this record