Load-sensitive selective pruning for distributed search

Broccolo, D., Macdonald, C. , Orlando, S., Ounis, I. , Perego, R., Silvestri, F. and Tonellotto, N. (2013) Load-sensitive selective pruning for distributed search. In: CIKM '13: 22nd ACM International Conference on Conference on Information and Knowledge Management, San Francisco CA, USA, 27 Oct - 1 Nov 2013, pp. 379-388. (doi: 10.1145/2505515.2505699)

93573.pdf - Accepted Version


Publisher's URL: http://dx.doi.org/10.1145/2505515.2505699


A search engine infrastructure must be able to provide the same quality of service to all queries received during a day. During normal operating conditions, the demand for resources is considerably lower than under peak conditions, yet an oversized infrastructure would result in an unnecessary waste of computing power. A possible solution adopted in this situation might consist of defining a maximum threshold processing time for each query, and dropping queries for which this threshold elapses, leading to disappointed users. In this paper, we propose and evaluate a different approach, where, given a set of different query processing strategies with differing efficiency, each query is considered by a framework that sets a maximum query processing time and selects which processing strategy is the best for that query, such that the processing time for all queries is kept below the threshold. The processing time estimates used by the scheduler are learned from past queries. We experimentally validate our approach on 10,000 queries from a standard TREC dataset with over 50 million documents, and we compare it with several baselines. These experiments encompass testing the system under different query loads and different maximum tolerated query response times. Our results show that, at the cost of a marginal loss in terms of response quality, our search system is able to answer 90% of queries within half a second during times of high query volume.

Item Type:Conference Proceedings
Additional Information:ISBN: 9781450322638
Glasgow Author(s) Enlighten ID:Macdonald, Professor Craig and Ounis, Professor Iadh
Authors: Broccolo, D., Macdonald, C., Orlando, S., Ounis, I., Perego, R., Silvestri, F., and Tonellotto, N.
College/School:College of Science and Engineering > School of Computing Science
Copyright Holders:Copyright © 2013 ACM
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher.
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record