Static Pruning for Multi-Representation Dense Retrieval

Acquavia, A., Tonellotto, N. and Macdonald, C. (2023) Static Pruning for Multi-Representation Dense Retrieval. In: 23rd ACM Symposium on Document Engineering (DocEng'23), Limerick, Ireland, 22-25 Aug 2023, ISBN 9798400700279 (doi: 10.1145/3573128.3604896)

[img] Text
300119.pdf - Accepted Version



Dense retrieval approaches are challenging the prevalence of inverted index-based sparse representation approaches for information retrieval systems. Different families have arisen: single representations for each query or passage (such as ANCE or DPR), or multiple representations (usually one per token) as exemplified by the ColBERT model. While ColBERT is effective, it requires significant storage space for each token's embedding. In this work, we aim to prune the embeddings for tokens that are not important for effectiveness. Indeed, we show that, by adapting standard uniform and document-centric static pruning methods to embedding-based indexes, but retaining their focus on low-IDF tokens, we can attain large improvements in space efficiency while maintaining high effectiveness. Indeed, on experiments conducted on the MSMARCO passage ranking task, by removing all embeddings corresponding to the 100 most frequent BERT tokens, the index size is reduced by 45%, with limited impact on effectiveness (e.g. no statistically significant degradation of NDCG@10 or MAP on the TREC 2020 queryset). Similarly, on TREC Covid, we observed a 1.3% reduction in nDCG@10 for a 38% reduction in total index size.

Item Type:Conference Proceedings
Additional Information:This work is supported, in part, by the spoke “Future HPC & BigData” of the ICSC – Centro Nazionale di Ricerca in High-Performance Computing, Big Data and Quantum Computing funded by European Union – NextGenerationEU, and the FoReLab project (Departments of Excellence).
Glasgow Author(s) Enlighten ID:Tonellotto, Dr Nicola and Macdonald, Professor Craig
Authors: Acquavia, A., Tonellotto, N., and Macdonald, C.
College/School:College of Science and Engineering > School of Computing Science
Research Centre:College of Science and Engineering > School of Computing Science > IDA Section > GPU Cluster
Copyright Holders:Copyright © 2023 ACM
First Published:First published in Proceedings of the 23rd ACM Symposium on Document Engineering (DocEng'23)
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record