Sen, P., Saha, S., Ganguly, D. , Verma, M. and Roy, D. (2022) Measuring and Comparing the Consistency of IR Models for Query Pairs with Similar and Different Information Needs. In: 31st ACM International Conference on Information and Knowledge Management (CIKM '22), Atlanta, Georgia, USA, 17-21 October 2022, pp. 4449-4453. ISBN 9781450392365 (doi: 10.1145/3511808.3557637)
Full text not currently available from Enlighten.
Abstract
A widespread use of supervised ranking models has necessitated an investigation on how consistent their outputs align with user expectations. While a match between the user expectations and system outputs can be sought at different levels of granularity, we study this alignment for search intent transformation across a pair of queries. Specifically, we propose a consistency metric, which for a given pair of queries - one reformulated from the other with at least one term in common, measures if the change in the set of the top-retrieved documents induced by this reformulation is as per a user's expectation. Our experiments led to a number of observations, such as DRMM (an early interaction based IR model) exhibits better alignment with set-level user expectations, whereas transformer-based neural models (e.g., MonoBERT) agree more consistently with the content and rank-based expectations of overlap.
| Item Type: | Conference Proceedings |
|---|---|
| Status: | Published |
| Refereed: | Yes |
| Glasgow Author(s) Enlighten ID: | Ganguly, Dr Debasis |
| Authors: | Sen, P., Saha, S., Ganguly, D., Verma, M., and Roy, D. |
| College/School: | College of Science and Engineering > School of Computing Science |
| ISBN: | 9781450392365 |
| Related URLs: |
University Staff: Request a correction | Enlighten Editors: Update this record
Deposit and Record Details
Deposit and Record Details