Query-Specific Variable Depth Pooling via Query Performance Prediction

Ganguly, D. and Yilmaz, E. (2023) Query-Specific Variable Depth Pooling via Query Performance Prediction. In: 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR23), Taipei, Taiwan, 23-27 July 2023, pp. 2303-2307. ISBN 9781450394086 (doi: 10.1145/3539618.3592046)

Abstract

Due to the massive size of test collections, a standard practice in IR evaluation is to construct a 'pool' of candidate relevant documents comprised of the top-k documents retrieved by a wide range of different retrieval systems - a process called depth-k pooling. A standard practice is to set the depth (k) to a constant value for each query constituting the benchmark set. However, in this paper we argue that the annotation effort can be substantially reduced if the depth of the pool is made a variable quantity for each query, the rationale being that the number of documents relevant to the information need can widely vary across queries. Our hypothesis is that a lower depth for queries with a small number of relevant documents, and a higher depth for those with a larger number of relevant documents can potentially reduce the annotation effort without a significant change in IR effectiveness evaluation. We make use of standard query performance prediction (QPP) techniques to estimate the number of potentially relevant documents for each query, which is then used to determine the depth of the pool. Our experiments conducted on standard test collections demonstrate that this proposed method of employing query-specific variable depths is able to adequately reflect the relative effectiveness of IR systems with a substantially smaller annotation effort.

• Item Type: Conference Proceedings
• Keywords: IR model evaluation, depth pooling, query performance prediction.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Ganguly, Dr Debasis
• Authors: Ganguly, D., and Yilmaz, E.
• College/School: College of Science and Engineering > School of Computing Science
• ISBN: 9781450394086
• Copyright Holders: Copyright © 2023 held by the owner/author(s)
• First Published: First published in SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval
• Publisher Policy: Reproduced in accordance with the publisher copyright policy