Static Pruning for Multi-Representation Dense Retrieval

Acquavia, A., Tonellotto, N. and Macdonald, C. (2023) Static Pruning for Multi-Representation Dense Retrieval. In: 23rd ACM Symposium on Document Engineering (DocEng'23), Limerick, Ireland, 22-25 Aug 2023, ISBN 9798400700279 (doi: 10.1145/3573128.3604896)

Abstract

Dense retrieval approaches are challenging the prevalence of inverted index-based sparse representation approaches for information retrieval systems. Different families have arisen: single representations for each query or passage (such as ANCE or DPR), or multiple representations (usually one per token) as exemplified by the ColBERT model. While ColBERT is effective, it requires significant storage space for each token's embedding. In this work, we aim to prune the embeddings for tokens that are not important for effectiveness. Indeed, we show that, by adapting standard uniform and document-centric static pruning methods to embedding-based indexes, but retaining their focus on low-IDF tokens, we can attain large improvements in space efficiency while maintaining high effectiveness. Indeed, on experiments conducted on the MSMARCO passage ranking task, by removing all embeddings corresponding to the 100 most frequent BERT tokens, the index size is reduced by 45%, with limited impact on effectiveness (e.g. no statistically significant degradation of NDCG@10 or MAP on the TREC 2020 queryset). Similarly, on TREC Covid, we observed a 1.3% reduction in nDCG@10 for a 38% reduction in total index size.

• Item Type: Conference Proceedings
• Additional Information: This work is supported, in part, by the spoke “Future HPC & BigData” of the ICSC – Centro Nazionale di Ricerca in High-Performance Computing, Big Data and Quantum Computing funded by European Union – NextGenerationEU, and the FoReLab project (Departments of Excellence).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Tonellotto, Dr Nicola and Macdonald, Professor Craig
• Authors: Acquavia, A., Tonellotto, N., and Macdonald, C.
• College/School: College of Science and Engineering > School of Computing Science
• Research Centre: College of Science and Engineering > School of Computing Science > IDA Section > GPU Cluster
• ISBN: 9798400700279
• Copyright Holders: Copyright © 2023 ACM
• First Published: First published in Proceedings of the 23rd ACM Symposium on Document Engineering (DocEng'23)
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher

Related URLs

• Organisation
• Publisher